Way back in the day I wrote about this issue, where the SCOM agent in some cases can consume above typical resource levels of memory, handles, etc. When this occurs – we will restart the agent to kill any “runaway” processes. Read about this here:
One of the things I have noticed, is that on many of my servers, these thresholds are being breached on a regular basis – mostly due to the monitoringhost.exe processes needing to use more than the default of 300mb of RAM (private bytes).
The issue is, that you will likely have NO idea this is happening. We don’t generate any alerts for this by default – we simply “fix the problem” by creating a state change, then running a response script to bounce the agent. The bad part about this, is you could have agents in a constant restart loop.
In SCOM 2012 – I still recommend making the following changes via overrides: Open the “Operations Manager > Agent Details > Agents by Version” view in the console:
Open health explorer for one of the agents – and here is an example of an agent that has been bouncing on a regular basis:
I recommend the following:
Private bytes monitors should be set to a default threshold of 629145600 (double the default of 300MB)
Handle Count monitors should be set to 15,000 (the default of 6000 is too low)
In addition, on each monitor:
Override Generate Alert to True (to generate alerts)
Override Auto-Resolve to false (even though default is false, this must be set, to keep from auto-closing these so you can see them and their repeat count)
Override Alert severity to Information (to keep from ticketing on these events)
Override EACH monitor, “for all objects of another class” and choose “Agent” class.
This is a good configuration:
As a refresher – this will be common on any monitored systems that discover a large number of instances – such as Exchange, DNS, SQL servers, SCVMM, large web servers, etc.