The new and improved guide on HealthService Restarts. Aka – agents bouncing their own HealthService

re: The new and improved guide on HealthService Restarts. Aka – agents bouncing their own HealthService

Kevin Holman — Tue, 01 Feb 2011 18:54:42 GMT

6500 agents - yeah... it sounds like your deductions and plan are solid. From a health perspective... I have never heard of issues when the process is consuming too much memory on the RMS... only unresponsive - and this manifests itself in the consoles not connecting, or config not generating, or unhealthy events which are alerted on.

re: The new and improved guide on HealthService Restarts. Aka – agents bouncing their own HealthService

Ted T Hacker — Tue, 01 Feb 2011 16:33:20 GMT

The monitor is looking at the "Process\Private Bytes" counter for the "HealthService.exe" process which on our RMS is steadily around 1.7GB. We have increased the threshold to 2GB which it has not exceeded for a few days, now. We have lots of agents (6500) and the RMS has 24GB. We have a bunch of MPs, but we have tuned them to keep the chatter down. AD is the noisiest, of course.

I am all for alerting if this value is unhealthy. It seems that the amount of memory the healthservice and the other key SCOM processes use is going to expand as demand increases, but I would really want to alert if the amount of memory they are using is causing problems. If we turn off this threshold, are we relying on other montiors to trigger when things get really bad?

This monitor makes sense for agents since it triggers the healthservice to get restarted. Agents should not use many local resources and restarting an agent's health service is not that disruptive.

Restaring the healthservice on an RMS is a diffferent issue. It is worth restarting if it is stuck, but not if it is just really busy.

It sounds like your recommendation is to turn off this threshold for at least the RMS. Maybe 1.5GB is OK for regular management servers. Right?

re: The new and improved guide on HealthService Restarts. Aka – agents bouncing their own HealthService

Kevin Holman — Fri, 28 Jan 2011 20:28:29 GMT

@Ted -

I dont hear that a lot. What is going over 1.5GB? Healthservice or Private bytes? On a large RMS with lots of memory - this wouldnt suprise me. It is very normal for the config and sdk to go over 2GB... I dont often see the Healthservice or Monitoringhost.exe's get that big though....

How many agents? How much ram in RMS? What are your primary MP's? I would have no issue bumping this up or even turning it off for your RMS.....

re: The new and improved guide on HealthService Restarts. Aka – agents bouncing their own HealthService

Ted T Hacker — Fri, 28 Jan 2011 20:08:50 GMT

Is the healthservice private bytes threshold of 1500 MB appropriate for the RMS of a large environment? My RMS is routinely going over the threshold and back under. Should I override the threshold to a larger number? What value? Is the normal not to contually exceed value a function of the number of agents or management servers ?

re: The new and improved guide on HealthService Restarts. Aka – agents bouncing their own HealthService

Paul B — Wed, 06 Jan 2010 18:26:42 GMT

Thanks for the suggestions. I just recently installed the lateset operations manager MP and it cleared up a slew of flopping agents. once again thanks for blogging.

re: The new and improved guide on HealthService Restarts. Aka – agents bouncing their own HealthService

Layne — Mon, 21 Dec 2009 20:59:53 GMT

Hi Kevin. I've been using these suggestions for a while now, and have noticed on our domain controllers the restart is launched, but I never get the corresponding 6062 event that the restart was successful. On all other agents I get both the restart and success events.

What can be done to troubleshoot why the agents try to restart but actually don't on our domain controllers?

re: The new and improved guide on HealthService Restarts. Aka – agents bouncing their own HealthService

rob1974 — Mon, 21 Dec 2009 20:47:19 GMT

The old overrides wouldn't functional. The new values would still apply. Deleting them would be less confusing though :)