Kevin Holman's OpsMgr Blog : Hotfix

OpsMgr 2007 SP1 cumulative rollup hotfix has shipped!

kevinhol — Mon, 09 Nov 2009 19:06:09 GMT

If you cannot or will not upgrade to OpsMgr 2007 R2 anytime soon – then this hotfix is for you!

Available at: http://support.microsoft.com/kb/971541

This updates OpsMgr 2007 SP1 to 6.0.6278.100. This is a rollup covering many new issues, plus most of the previously released critical hotfixes for OpsMgr. I recommend this rollup hotfix for anyone running OpsMgr 2007 SP1 that doesn't have very near term plans in place to upgrade to OpsMgr 2007 R2.

Overview

The Update Rollup for Operations Manager 2007 Service Pack 1 (SP1) combines previous hotfix releases for SP1 with additional fixes and support of SP1 roles on Windows 7 and Windows Server 2008 R2. This update also provides database role and SQL Server Reporting Services upgrade support from SQL Server 2005 to SQL Server 2008.
The Update Rollup includes updates for the following Operations Manager Roles:

Root Management Server, Management Server, Gateway Server
Operations Console
Operations Management Web Console Server
Agent
Audit Collection Server (ACS Server)
Reporting Server

The following tools and updates are provided within this update which may be specific to a scenario:

Support Tools folder – Contains SRSUpgradeTool.exe and SRSUpgradeHelper.msi (Enables upgrade of a SQL Server 2005 Reporting Server used by Operations Manager Reporting to SQL Server 2008 Reporting Server)
Gateway folder – Contains a MSI transform and script to update MOMGateway.MSI for successful installation on Windows Server 2008 R2
ManagementPacks folder – Contains an updated Microsoft.SystemCenter.DataWarehouse.mp which requires manual import

Feature Summary:

Providing a rollup that supersedes nearly all SP1 binary hotfixes in a single package (~50 fixes) . See KB971541 for exceptions.
Support for Windows 7 and Windows Server 2008 R2 - See KB974722 which will be updated to include data around the release of KB971541
Operational and DataWarehouse database support for upgrade to SQL 2008.
Additional stability hotfixes
SCCM Monitoring via our 64-bit agent in the latest SCCM MP. See the latest SCCM MP guide for details.
Exchange 2010 MP support
Fix for Ops console crashes seen on Vista and Windows 7

The Supported Configurations Guide and Upgrade Guide have also been updated.

My experience upgrading a lab management group to the SP1 rollup:

First – I’d recommend making a plan…. like reading the KB article, known issues, and plan out the order of update operations. The KB article dont specifically state a specific order, so I will probably do something like this:

Root Management Server (includes web console)
all secondary Management Servers (includes any that are audit collectors)
SCOM Reporting Server
any stand alone OpsMgr consoles
Agents (both from pending and manually installed)

Ok – getting started…..

COPY the SystemCenterOperationsManager2007-SP1-KB971541-X86-X64-IA64-ENU.MSI file locally to the RMS.
Run the update from the MSI – install to default locations
If executing on Windows Server 2008 – run the MSI from an elevated command prompt.
READ the release notes (you can copy these out to word to make them more readable
Note the new splash screen for this hotfix:

We will “Run Server Update”
The hotfix installs with no further user interaction.
The installer will finish – you can click “Finish”. However – another installer will kick off immediately afterward. This is by design – documented in the release notes, and is for installing localization updates. Then click “Finish” on the second update screen.
At this point, you can click “Exit” on the Software Update splash screen.
Continue applying the updates to the different roles – as documented in the release notes.

Some interesting things you might notice:

We actually clean up all the old hotfixes from the agent files – and move them to the root of \AgentManagement folder:

This is good – we wont try and re-apply them to subsequent agent installs/updates. Now – there will be two new hotfix files in the \x86, \AMD64, and \ia64 folders:

These include the agent hotfix update, plus a localization update which will vary based on your localization settings.

As for double-checking the update applied successfully – you can add “File Version” column to windows Explorer:

You will notice several management packs got updated in the console:

***Note – per the release notes and KB article

Import the following management pack from the ManagementPacks folder:

Microsoft.SystemCenter.DataWarehouse.Reports.MP

The location is a bit confusion – the full path would be: \Program Files\System Center 2007 Hotfix Utility\KB971541\ManagementPacks\

You can import this at any time… I’d recommend importing this after you are done updating all the server roles, including reporting.

In the console – you will also note that any agents that were not manually installed – will require an agent update. I would hold off updating any agents until your management group server roles are fully updated, and then only update around 200 agents at a time. This process will cause significant database and management server activity – so I’d advise doing the agent updates during off-peak use hours for large management groups:

Next – I update my management servers, including any that run ACS (have a special ACS update for those too in the Hotfix installer splash screen)

Next up – reporting:

I kick this off on the SRS/SCOM Reporting server. Not prompted for anything… it completes in seconds.

Next on the list – update agents. They all updated just fine via pending actions except for one…. this happens to be the same server that is hosting the OpsDB. Interesting. I got the following error:

The MOM Server failed to perform specified operation on computer DB.opsmgr.net.

Operation: Agent Install

Install account: OPSMGR\localadmin

Error Code: 8007064A

Error Description: The configuration data for this product is corrupt. Contact your support personnel.

This turned out to be due to some bad data in my HKEY_CLASSES_ROOT\Installer\Products\C9A0067E2876122489E4BA987C08CDD2\Patches\Patches REG_MULTI_SZ value. I am not sure how this got messed up – probably due to me testing a bunch of SP1 hotfixes previously and fat fingering a registry edit – so I would not assume this will be a common error. Once I fixed this registry entry – this agent updated just fine as well.

How can I be sure all my agents got updated?

One of the simplest ways – is to look at the “Patchlist” column on a Health Service State view. Create a new State View in “My Workspace”. Target “Health Service”:

On the Display Tab – select only Name, and Patch List:

Voila!

Dude. Got R2? Then go get this update!

kevinhol — Fri, 09 Oct 2009 06:17:23 GMT

Seriously.

Operations Manager 2007 R2 Management Pack version 6.1.7533.0 from the MP CATALOG.

This is basically a set of the core MP’s in OpsMgr R2. You should expect to see these get updated on a fairly regular basis now. These are the MP’s that are built in to OpsMgr, that monitor the health of the Management group, from the server role, and agent role perspective. There should be a SP1 version of this update available soon to follow. These are a series of updates to OpsMgr, based on Community, Customer, and internal feedback…. and it’s good stuff.

Everyone running R2 should run these through your standard testing cycle and get these implemented. This is a simple MP import-update.

The MOMTEAM Blog has some more details here: http://blogs.technet.com/momteam/archive/2009/10/07/opsmgr-2007-r2-mp-version-6-1-7553-0-is-released.aspx

This includes a long list of updates (available in the guide in the download)… but I will hit a few high points.

You might have seen some of my blog posts on agents restarting all the time, in SP1 and R2, and the significant impact that can have:

http://blogs.technet.com/kevinholman/archive/2009/06/22/health-service-and-monitoringhost-thresholds-in-r2-how-this-has-changed-and-what-you-should-know.aspx

Well, this is mostly resolved in this update…. as the default threshold for HealthService and MonitoringHost has been changed from 100MB to 300MB. This will stop the majority of your agents from hitting this limit, and restarting. Once you import this MP update – you should review any overrids you made on these Monitors… and make sure you dont have any conflicts. The only other override I like to recommend is to generate an alert on these monitors – so that you will know when there is a problem that will cause the agent to get restarted – specifically on HealthService or MonitoringHost, and PrivateBytes or HandleCount:

You might find that there are still some agents that need a much higher threshold…. but those should be fairly rare, and limited to very large servers with a very high instance count of objects (large SQL clusters, large Exchange servers, etc) and also possibly for agents that are a watcher node or proxy for a large number of devices (network, VMWare, etc…)

In THIS blog post I discussed adding a custom perf counter collection rule and creating a view to see how many consoles or SDK connections you have. This is now built-in:

Lots of Knowledge and descriptions added to help troubleshoot alerts. For instance – on the common “Script or Executable Failed to run”

Summary

The Health Service was attempting to run an executable and was unable to create the process. This may affect some monitoring or discovery.

Causes

This can be caused by:

The executable could not be found.

The computer does not have enough resources (for example; memory) to run the executable.

The antivirus software on the computer blocking Visual Basic scripts or Java scripts. The following link is a link to the KB article regarding this issue.

Antivirus software blocking script execution

Resolutions

The alert description and context has information indicating which rule or monitor failed. The following link will display all events indicating a failure to run the executable:

View Batch Response Events

After reviewing the error in the context, check:

That the path to the executable exists on the computer.

The antivirus software is not blocking scripts from running.

That the computer is not over utilized.

Check Task Manager to see if there is enough free memory.

Check Task Manager to see if there are any processes consuming all the CPU.

Check the error information in the event or alert context for the script path and name. There could be a problem with the script not handling an error correctly and exiting. If the script exits without outputting the data that is expected (e.g. property bag data), this error is raised.

Another error can be caused by the misconfiguration of the workflow executing this script. The configuration (script params, policy, timeout) could be wrong causing no output or timeout.

There is a big list of other enhancements or fixes in the MP guide. Don't take this lightly – this is good stuff…. get it updated.

Fixing troubled agents

kevinhol — Fri, 02 Oct 2009 00:23:46 GMT

Sometimes agents either will not “talk” to the management server upon initial installation, and sometimes an agent can get unhealthy long after working fine. Agent health is an ongoing task of any OpsMgr Admin’s life.

This post in NOT an “end to end” manual of all the factors that influence agent health…. but that is something I am working on for a later time. There are so many factors in an agent’s ability to communicate and work as expected. A few key areas that commonly affect this are:

DNS name resolution (Agent to MS, and MS to Agent)
DNS domain membership (disjointed)
DNS suffix search order
Kerberos connectivity
Kerberos SPN’s accessible
Firewalls blocking 5723
Firewalls blocking access to AD for authentication
Packet loss
Invalid or old registry entries
Missing registry entries
Corrupt registry
Default agent action accounts locked down/out (HSLockdown)
HealthService Certificate configuration issues.
Hotfixes required for OS Compatibility
Management Server rejecting the agent

How do you detect agent issues from the console? The problem might be that they are not showing up in the console at all! Perhaps they might be a manual install that never shows up in Pending Actions? Or a push deployment, that stays stuck in Pending actions and never shows up under “Agent Managed”. Or even one that does show up under “Agent Managed” but never shows as being monitored… returning agent version data, etc.

One of the BEST things you can do when faced with an agent health issue… if to look on the agent, in the OperationsManager event log. This is a fairly verbose log that will almost always give you a good hint as to the trouble with the agent. That is ALWAYS one of my first steps in troubleshooting.

Another way of examining Agent health – is by the built in views in OpsMgr. In the console – there is a view – Located at the following:

This view is important – because it gives us a perspective of the agent from two different points:

1. The perspective of the agent monitors running on the agent, measuring its own “health”.

2. The perspective of the “Health Service Watcher” which is the agent being monitored from a Management Server".

If any of these are red or yellow – that is an excellent place to start. This should be an area that your level 1 support for Operations manager checks DAILY. We should never have a high number of agents that are not green here. If they aren't – this is indicative of an unhealthy environment, or the admin team not adhering to best practices (such as keeping up with hotfixes, using maintenance mode correctly, etc…

Use Health Explorer on these views – to drill down into exactly what is causing the Agent, or Health Service Watcher state to be unhealthy.

Now…. the following are some general steps to take to “fix” broken agents. These are not in definitive order. The order of steps really comes down to what you find when looking at the logs after taking these steps.

Start the HealthService on the agent. You might find the HealthService is just not running. This should not be common or systemic. Consider enabling the recovery for this condition to restart the HealthService on Heartbeat failure. However – if this is systemic – it is indicative of something causing your HealthService to restart too frequently, or administrators stopping SCOM. Look in the OpsMgr event log for verification.

Bounce the HealthService on the agent. Sometimes this is all that is needed to resolve an agent issue. Look in the OpsMgr event log after a HealthService restart, to make sure it is clean with no errors.

Clear the HealthService queue and config (manually). This is done by stopping the HealthService. Then deleting the “\Program Files\System Center Operations Manager 2007\Health Service State” folder. Then start the HealthService. This removes the agent config file, and the agent queue files. The agent starts up with no configuration, so it will resort to the registry to determine what management server to talk to. From the registry – it will find out if it is AD integrated, or a fixed management server to talk to if not. This is located at HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Agent Management Groups\PROD1\Parent Health Services\ location, in the \<#>\NetworkName string value. The agent will contact the management server – request config, receive config, download the appropriate management packs, apply them, run the discoveries, send up discovery data, and repeat the cycle for a little while. This is very much what happens on a new agent during initial deployment.

Clear the HealthService queue and config (from the console). When looking at the above view (or any state view or discovered inventory view which targets the HealthService or Agent class) there is a task in the actions pane - “Flush Health Service State and Cache”. This will perform a very similar action to that above…. as a console task. This will only work on an agent that is somewhat responsive…. if it does not work you need to perform this manually as the agent is really broken from communication with the management server. This task will never complete, and will not return success – because the task breaks off from itself as the queue is flushed.

“Repair” the agent from the console. This is done from the Administration pane – Agent Managed. You should not run a repair on any AD-integrated agent – as this will break the AD integration and assign it to the management server that ran the repair action. A “repair” technically just reinstalls the agent in a push fashion, just like an initial agent deployment. It will also apply/reapply any agent related hotfixes in the management server’s \Program Files\System Center Operations Manager 2007\AgentManagement\ directories.

Reinstall the agent (manually). This would be for manual installs or when push/repair is not possible. This section is where the combination of options gets a little tricky. When you are at this point… where you have given up, I find just going all the way with a brute force reinstall is the best way. This means performing the following steps:

Uninstall the agent via add/remove programs.
Run the Operations Manager Cleanup Tool CleanMom.exe or CleanMOM64.exe. This is designed to make sure that the service, files, and all registry entires are removed.
Ensure that the agent’s folder is removed at: \Program Files\System Center Operations Manager 2007\
Ensure that the following registry keys are deleted:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HealthService

Reboot the agent machine (if possible)
Delete the agent from Agent Managed in the OpsMgr console. This will allow a new HealthService ID to be detected and is sometimes a required step to get an agent to work properly, although not always required.
Now that the agent is gone cleanly from both OpsMgr console and the agent Operating System…. manually reinstall the agent. Keep it simple – install it using a named management server/management group, and use Local System for the agent action account (these will remove any common issues with a low priv domain account, and AD integration if used) If it works correctly – you can always reinstall again using low priv or AD integration.
Remember to import certificats at this point if you are using those on the individual agent.
As always – look in the OperationsManager event log…. this will tell you if it connected, and is working, or if there is a connectivity issue.

To summarize…. there are many things that can cause an agent issue, and many methods to troubleshoot. However – to summarize at a very general level, my typical steps are:

Review OpsMgr event log on agent
Bounce HealthService
Bounce HealthService clearing \Health Service State folder.
Complete brute force reinstall of the agent.

If it an external issue is causing the issue (DNS, Kerberos, Firewall) then these steps likely will not help you…. but those should be available from the OpsMgr event log.

Also – make sure you see my other posts on agent health and troubleshooting during deployment:

Console based Agent Deployment Troubleshooting table

Agent discovery and push troubleshooting in OpsMgr 2007

Getting lots of Script Failed To Run alerts- WMI Probe Failed Execution- Backward Compatibility

Agent Pending Actions can get out of synch between the Console, and the database

Which hotfixes should I apply-

Do you randomly see a MonitoringHost.exe process consuming lots of CPU?

kevinhol — Mon, 20 Jul 2009 18:14:24 GMT

Randomly, you might see a single MonitoringHost.exe process on an agent, consuming 100% CPU. (Or 50%, or 25% depending on how many cores you have). This process will stay at this level, and will not recover. If you restart the OpsMgr HealthService, the problem goes away, and might not return for days or even weeks.

This particular symptom, might be due to an XML spinlock issue… this is a core Windows OS issue, and there is a hotfix available, which I have on my HOTFIX LINK

The KB is 968967 :

“The CPU usage of an application or a service that uses MSXML 6.0 to handle XML requests reaches 100% in Windows Server 2008, Windows Vista, Windows XP Service Pack 3, or other systems that have MSXML 6.0 installed”

I have seen that most customers are affected by this issue from time to time. I have seen it very commonly in my lab, on Server 2008 Domain controllers, and my Server 2008 Hyper-V hosts…

A note on patching Server 2008:

When you go to download this hotfix for a server 2008 machine – it is very misleading on which hotfix to even get. Here is the list of all available fixes:

For patching Server 2008 – you need to download the “Windows Vista” hotfix – in either x86 or x64, depending on your OS version:

Monitoring for this condition:

You can easily write a threshold monitor targeting agent or HealthService, to track the monitoringhost process \ %processor time threshold, and set it to alert when it has multiple consecutive samples above a defined threshold.

Here is an example of creating this monitor:

Authoring Pane > Monitors > New Unit Monitor > Windows Performance Counters > Static Thresholds > Single Threshold > Consecutive Samples over Threshold.

Give it a custom name that follows your documented custom Monitor naming standard, target “Health Service”, and put this under Performance rollup.

Hit the “Select” button (in SP1 – select “Browse”) In the perf counter picker – choose a server with an installed agent, choose the Object “Process” the counter “%Processor Time” and the Instance “MonitoringHost”, and click OK.

Since there are multiple MonitoringHost processes… we will add a Wildcard to the Instance name in the monitor…. this will monitor ANY MonitoringHost process for high CPU. Set the Interval to every 1 minute.

For the number of consecutive samples, and threshold… that is up to you. For me – I will say that if I detect a single MonitoringHost process using more than 50% CPU, over all 5 consecutive samples (5 minutes) then I consider that bad:

At this point…. you can simply alert on the condition, or event try and add a recovery script – that will bounce the health service. Generally, bouncing the HealthService when one of the processes is using all the CPU is not always 100% reliable… especially from a “NET STOP & NET START” type command. I have found it more reliable to just kill the MonitoringHost process in this condition, and allow it to respawn…. but your mileage may vary.

http://blogs.technet.com/kevinholman/archive/2008/03/26/using-a-recovery-in-opsmgr-basic.aspx

New Dell Hardware MP released, 3.1.1 A00

kevinhol — Fri, 27 Mar 2009 03:09:54 GMT

Dell has updated their hardware management pack. The old version was 3.1.0.118. The new version is 3.1.1.3. I blogged about the old MP HERE

So – first thing – I did the unthinkable. I read the release notes. Here is what Dell says is new:

#####################################################################
RELEASE HIGHLIGHTS
#####################################################################

* A known defect with storage component discovery, where in some
scenarios (especially with Server Administrator version 5.5 or above
installed),storage details could not be viewed under servers in the
diagram view of the SCOM console, has been fixed.(DF283684/DF257743)

* A defect with error handling in the DellBMCLog.exe while handling
DellBMCLog error events in a Distributed Management Server
environment has been fixed.(DF260495)

* A known defect with health of Virtual Disks, where in some
scenarios health of Virtual Disks were not shown correctly, has been
fixed.(DF279394)

At first glance – I can see the PDF guide has not been updated. So no major changes there.

Bummer – Server 2008 is still not supported, according to the release notes and guide. :-(

If you are going to leave the SNMP stuff enabled – you absolutely still need to make sure you have KB 951526 installed – or a newer KB that updates the same file - Momnetworkmodules.dll…. and the most current hotfix KB for that is 957511

I can see that DellBMCLogSetup.exe has been updated… this should be the fix referenced above… where this tool would crash on all your management servers previously.

I can see that Dell_MDStorageArray_ABBSetup.exe has been updated… so I assume this has something to do with the fixes in storage component discovery, or virtual disks.

I can see all of the MP files have been updated with a new time/date stamp as well.

Next, I ran a MPDiff to see what's new in this release.

1. First off – there are 62 differences. The majority of these are a change in the discoveries – to change the “MinLength” value from “1” to “0”. The reason for this… is because if a discovery is set with a “1” or greater, the discovery will not work correctly if the discovered value is NULL. Essentially, some of these discovery properties probably are empty on some systems, and the fact they were set to “1” previously caused discovery data failures on agents. You can read more about this here: http://blogs.technet.com/smsandmom/archive/2008/09/09/opsmgr-2007-discovery-data-processing-fails-with-invalid-monitoring-class-property-value-message.aspx

2. Next - DellStorageDiscovery.vbs script was updated with a few lines of code.

3. Next - DellStorageControllerUnitMonitor.vbs arguments and script were updated a bit.

4. Next - ControllerVirtualDiskEventTrigger.vbs arguments and script were updated a bit.

#2, 3, and 4 above all appear to relate to fixing the storage discovery, and Virtual Disk issue documented in the release notes.

So….. not much as far as changes go. The rest of this post – I will talk about the most common changes I make in this MP.

So…. first – to really understand any management pack - I run MPViewer against the MP. This will help me to understand the classes, tasks, dependencies, discoveries, rules, monitors, etc…

The first thing I look at – is the Object discoveries. I want to know:

1. What are we discovering?

2. How often are we discovering it?

3. What is the discovery “hierarchy”…. meaning… what is the base discovery, and what other discoveries run against the target instances of the bas discovery. This allows us to chain together, how the MP works… and will help us troubleshoot if something isn't showing up.

(See my previous post – where I also talk about these discoveries, what they do, how they work)

Here is the MPviewer discovery output:

The core discovery is running against all “Windows Computers” every 6 hours.

Using MPviewer – I can see the core XML for this discovery:

What this tells me – is that this discovers an object class, called “DellSystem”. That’s my first discovered class type. I can look at this in the monitoring console - discovered inventory:

Then – I can look above at the MPViewer output – and see that “Dell Server Discovery” targets “Dell Systems Instances”. Using the same methods – you can see the subsequent discoveries that target “Dell Server Instance” created by this discovery, and so on, and so on. Why is this important? It helps us understand the MP, and helps us find out when a MP isn’t “finding” something that we think it should.

Ok…. so, in general, here is what I turn off in this MP.

First – this MP includes Dell Printer discovery and monitoring. Let’s say that I am not interested in monitoring printers, and don't want the additional load created by this? So – I disable the “Dell printer Discovery”.

I also decide – I don't want to deploy the SNMP based DRAC/CMC based moniotring…. or any SNMP based stuff for that matter…. so I disable all the discoveries that are targeted to “Microsoft.SystemCenter.Network.Device”

Continuing on…. I look at Monitors, and disable all the Monitors for the same:

Next up – rules…. I disable all the SNMP based rules that are targeting “Dell.Connections.CMC, Dell.Connections.DRAC_5, and Dell.Connections.DRACMC”…. consistent with my “No SNMP” approach.

Lastly on rules – since I decided not to implement the BMCLog executables on my management servers… I disable the two rules that are running on my Management Servers.

Lastly – and probably MOST important…. If I have a LARGE agent count…. these discoveries put a lot of data in the database. If they run too frequently, AND collect discovery properties that change often…. I will fill my OpsDB and Warehouse with ManagedEntityProperty data. This may not sound significant – but it is. A management pack should NOT collect information as a discovery property of a class, if the information will change often. For instance, The “Temperature Unit Instance” class, contains a discovered property, called “Sensor Current Reading”

I am just picking on this one as an example…. this is the current reading of the temperature sensor… and probably changes a bit between each discovery. This means every 6 hours (and much more frequent in previous versions of the older Dell MP’s) this data is added to the database…. and this builds up over time. It is much better to create a monitor/rule for this – and enter this data into the database as performance data via a propertybag, instead of using a discovery property on a class.

For this reason – I like to decrease the frequency of discoveries if this is a known risk. So – using my MPViewer data – I am off to the console – to find these discoveries, and change their frequency from every 6 hours, to every 12 or 24 hours…. depending on how fast you *need* your Dell systems to be discovered once they get an agent installed.

The problem is, the first discovery I want to look at, the “Dell System Discovery” is nowhere to be found in the console! This is because they did not define a discovery type. This is required for the Object Discovery to be viewable in the console. It’s weird – because some of their discoveries DO show up… it looks like they just left this off some of them.

Much like the IBM MP I blogged about here: http://blogs.technet.com/kevinholman/archive/2009/02/27/issues-with-the-current-ibm-mp-2-0-0-501-and-a-workaround.aspx We will use the same workaround to get to all the Dell discoveries.

So – using the workaround, I search for Object Discoveries using “Dell” as the keyword:

From here – I can click “View Knowledge” to gain access to the discovery rule in the UI – and create an override to run this every 12 or 24 hours… depending on what you want. I can also use this to disable the discoveries I mentioned above if you elect not to use them.

Now…. we are ready to tune the MP… there are a LOT of alerts generated in the field, in my experience, from items that not everyone considers valuable, like voltage sensors, etc… so get ready to turn off some alert rules as well, based on your monitoring requirements.

Applying an OpsMgr hotfix to a RMS Cluster node? Some things to be aware of.

kevinhol — Wed, 25 Feb 2009 05:20:09 GMT

When you apply a SCOM hotfix to a RMS cluster, you need to be aware of some issues, and some workarounds. This is something I have seen several times in the field…

On any server/agent, the Hotfix installer will stop any discovered OpsMgr services, including the SDK, Config, and HealthService. This part is normal. It does this in order to update the files (DLL’s) that are part of the hotfix payload, and then it will start the services again when complete. This all works well, except for on RMS clusters.

The reason for this, is that the Hotfix installer is not 100% cluster aware.

In a RMS cluster… the passive node will have these three services stopped, and the services will be set to Manual Startup. On the active node – the OpsMgr services are also set to Manual Startup, but the services are running, because the Cluster service controls these services now. This is how a clustered service works, and we should not ever stop a clustered service in Service Control Manager, we really should take the resource offline, in Cluster Admin.

So I have two options… I can apply the hotfix to the Active Node… or the Passive node.

If I choose the active node – the hotfix installer will try and stop all the OpsMgr services, and this will cause the Cluster service to try and restart them, or eventually fail them over to the passive node – depending on your Cluster configuration settings. Therefore – it is probably best to patch the passive node first… ensure the hotfix applied correctly, and then move the cluster group and OpsMgr RMS group over to the freshly hotfixed node… and go patch the other one (now passive)

This works – but is not 100% smooth. When we apply the hotfix to the passive node, the hotfix installer will try and start the services at the end of the process, even though they were not running previously. We do NOT want these services trying to run on the passive node – since it does not own the cluster disk resources…. so the services will start, but cannot do anything but log errors.

You will also see an error from the HealthService – not being able to start. It is apparent that this service fails because it cannot access the disk resource, but the SDK and config services WILL start.

What is worse – is that the hotfix installer – changes the config of the service startup types to Automatic – which means these services will continue to try and run on the passive node across reboots.

So – the guidance I have, for RMS clusters – is:

Patch the passive node (we will call this Node 2)
Click ok on the HealthService start failure error.
Ensure the hotfix applied by inspecting the DLL(s) versions as documented in the KB.
Stop the running SDK and Config services on the passive node.
Set any OpsMgr services that were changed to Automatic – BACK to Manual.
Move the cluster resource groups over to the freshly patched Node 2.
On Node 1 (now passive) apply the hotfix, and repeat steps starting at Step 2 above.

NOTE: This is only applicable to OpsMgr specific hotfixes. For OS hotfixes – you would follow your standard clustered OS hotfix routine.

New OpsMgr hotfix – 958253 – Fixes patchlist table view for agent hotfixes

kevinhol — Tue, 27 Jan 2009 20:45:35 GMT

Hey guys – a new hotfix is published, and I recommend this for all SP1 environments. This cleans up the Patchlist table in the database, making it more readable, and allowing more room because of the 255 character limit.

You can now more easily know which hotfixes have been applied to which agents. This is a simple MP update – just import it and go.

http://support.microsoft.com/kb/958253

I discuss the process to know that your agents are patched" in the following posts:

http://blogs.technet.com/kevinholman/archive/2008/06/24/how-do-i-know-which-hotfixes-have-been-applied-to-which-agents.aspx

http://blogs.technet.com/kevinholman/archive/2008/06/27/a-report-to-show-all-agents-missing-a-specific-hotfix.aspx

Before – the patchlist table looked like this:

Now – it is much more readable:

I have added this one to my recommended list, at http://blogs.technet.com/kevinholman/archive/2009/01/27/which-hotfixes-should-i-apply.aspx

Which hotfixes should I apply?

kevinhol — Tue, 27 Jan 2009 17:19:38 GMT

This is updated as of 11-9-2009 (added SP1 rollup hotfix)

In general - you should evaluate all hotfixes available, and only apply those applicable to your environment. However, some of these below I have seen impact almost every environment, and should be heavily considered.

This list is nothing official.... this is just a general list of the recommended hotfixes I end up proactively applying to most environments.... it is not a complete list of ALL hotfixes, and you may be affected by other issues.

Before we get to the lists – some general guidance on hotfixes to make you more successful:

ALWAYS - on Server 2008 OS, run the hotfix MSI from an elevated command prompt window. This will launch the install of the hotfix, and then launch the boot-strapper window in an elevated process – which is required. Do this regardless of the UAC configuration of the 2008 OS.

ALWAYS - make sure you read the instructions to understand if the hotfix is a SQL update, installed to the RMS, MS, and/or Gateway, AND/OR applies to agents as well.

ALWAYS - make sure you double-check the DLL version of the updated files to make sure the hotfix successfully applied after installing.

ALWAYS - make sure you double-check the \AgentManagement directory of the management servers and gateways, to make sure if there is an agent update, the x86 and x64 MSP was copied over correctly.

ALWAYS – when installing a hotfix on a SCOM server role, run the downloaded MSI, such as “SystemCenterOperationsManager2007-SP1-KB954049-X86-X64-ENU.MSI” – and install the “System Center 2007 Hotfix Utility” to the DEFAULT location – and then kick off the update FROM THE UI that comes up by clicking “Run Software Update”. This is critical and not following this process is the cause for many failures to apply the hotfix DLL’s, or failure to copy the agent MSP update files to the \Agentmanagement directory. NEVER run the MSP files manually on a SCOM server role… because the additional steps run by the boot-strapper will not execute if you do that.

ALWAYS check the language version of the hotfix, and make sure it is the same language version as your SCOM base install. For instance – if you have a English base SCOM install – do not download a localized German version of a hotfix and apply it – or it can break the English SCOM base install.

Common OpsMgr 2007 Post-R2 hotfixes:

This list ABSOLUTELY assumes you are at OpsMgr R2-RTM level as a base (6.1.7221.0).

Hotfix	Update Files	Supersedes Hotfix(s)	Resolves	Applies to:	Comments
MP Update	Microsoft.SystemCenter.2007.mp 6.1.7533.0 Microsoft.SystemCenter.OperationsManager.2007.mp 6.1.7533.0 Microsoft.SystemCenter.OperationsManager.AM.DR.2007.mp 6.1.7533.0	none	Agent restarts, other enhancements	MP import only	I recommend this update for ALL OpsMgr R2 environments.
971233	none	none	The console shows customized subscriptions SMTP{`GUID`} after you upgrade to OpsMgr R2 from OpsMgr SP1	Operations Database (TSQL only)	I recommend this hotfix only if you are impacted with this issue.

Common OpsMgr 2007 Post-SP1 hotfixes:

This list ABSOLUTELY assumes you are at OpsMgr SP1 level as a base (6.0.6278.0). These DO NOT APPLY these to OpsMgr R2.

Hotfix	Update Files	Supersedes Hotfix(s)	Resolves	Applies to:	Comments
MP Update	Microsoft.SystemCenter.2007.mp 6.0.6709.0 Microsoft.SystemCenter.OperationsManager.2007.mp 6.0.6709.0 Microsoft.SystemCenter.OperationsManager.AM.DR.2007.mp 6.0.6709.0	none	Agent restarts, many other critical enhancements	Management Pack Import only (Import via console once extracted)	I recommend this update for ALL OpsMgr SP1 environments.
971541	SP1 Rollup hotfix. Multiple files. See KB article	Many. See KB article	Many. See KB article	RMS MS GW Reporting Agents	I recommend this update for ALL OpsMgr SP1 environments.
954643	Managementpackinstall.sp.sql	none	Event ID 31569 is logged after you install a management pack that includes reports on a System Center Operations Manager 2007 SP1 server	Data Warehouse Database (T-SQL only)	This hotfix includes a SQL script, which you execute on the database in a query window. I recommend this hotfix only if you are impacted with these events.

Common related Windows Operating System Hotfixes:

This list is not sorted by OS or anything special – just a collection of OS related hotfixes that SCOM might require, or might fix an issue with the OS that impacts OpsMgr. These can apply to Sp1 or R2 environments.

Hotfix	Update Files	Supersedes Hotfix(s)	Resolves	Applies to:	Comments
968760	See article	N/A	High handle count on the RMS A managed application has a high number of thread handles and of event handles in the Microsoft .NET Framework 2.0	RMS	I recommend this hotfix is you are experiencing high handle count on the RMS. This hotfix requires SP2 for the OS and .NET 2.0 SP2.
968967	See article	N/A	The CPU usage of an application or a service that uses MSXML 6.0 to handle XML requests reaches 100% in Windows Server 2008, Windows Vista, Windows XP Service Pack 3, or other systems that have MSXML 6.0 installed (Spinlock)	RMS MS GW Agent	I recommend this hotfix if you are impacted with this issue. You might find a MonitoringHost.exe process randomly stuck at 100% CPU. If so – this hotfix might be applicable
951327	See article	N/A	The System Center Operations Manager 2007 console may crash in Windows Server 2008 or in Windows Vista when you open the Health Explorer window	Any Vista or Server 2008 computer with a SCOM console installed	I recommend this hotfix only if you run the console on Server 2008 or Vista. This hotfix is already included in Server 2008 SP2.
952664	See article	N/A	The Event Log service may stop responding because of a deadlock on a Windows Server 2008-based or Windows Vista-based computer	RMS MS GW Agent	I recommend this hotfix only if you host an OpsMgr server or agent role on Vista or Server 2008. This hotfix is already included in Server 2008 SP2.
953290	See article	N/A	An application may crash when it uses legacy methods to query performance counter values in Windows Vista or in Windows Server 2008	RMS MS GW Agent	I recommend this hotfix only if you host an OpsMgr server or agent role on Vista or Server 2008. This hotfix is already included in Server 2008 SP2.
958661	See article	N/A	FIX: Small memory leaks may occur when you use RSCA to query runtime statistics in IIS 7.0	Any OpsMgr Agent/Server role with IIS 7.0 installed	I recommend this hotfix in all cases where you are monitoring servers with IIS 7.0 installed, and use the IIS Management pack.
958807	See article	N/A	Windows Server 2008 Failover Clustering WMI provider does not correctly handle invalid characters in the private property names causing WMI queries to fail	Any Server 2008 agent managed cluster node	I recommend this hotfix only if you are impacted with this issue, and use the current Cluster MP.

Make sure you see these additional posts on the subject of hotfixes:

http://blogs.technet.com/kevinholman/archive/2008/06/25/a-little-tidbit-on-hot-fixes-for-opsmgr.aspx

http://blogs.technet.com/kevinholman/archive/2008/06/24/how-do-i-know-which-hotfixes-have-been-applied-to-which-agents.aspx

http://blogs.technet.com/kevinholman/archive/2008/06/27/a-report-to-show-all-agents-missing-a-specific-hotfix.aspx

http://blogs.technet.com/kevinholman/archive/2009/02/25/applying-an-opsmgr-hotfix-to-a-rms-cluster-node-some-things-to-be-aware-of.aspx

Several of these updates require agent updates as well, so be prepared to deal with those.

Installing the Web Console on a 2008 Management Server - using Windows Authentication

kevinhol — Wed, 24 Sep 2008 21:53:22 GMT

Below is a step by step on taking a Windows 2008 Management Server, and adding the Web Console... with the requirement of using Windows authentication. The easiest method is to use Forms Based auth for Web Console servers.... but using Windows Auth is possible if you can leverage constrained delegation (more on this later).

I will start by running setup, and checking the prerequisites for the web console:

We need to add the Web Server Role, and make sure we include all required sub roles. This is documented here:

http://blogs.technet.com/kevinholman/archive/2008/09/26/how-to-install-iis-on-server-2008-to-support-opsmgr-web-console-and-reporting.aspx

Once IIS is installed correctly - now run the pre-requisite check again:

All good. At this point - we can run SetupOM.exe, and add the web console component.

We will choose Windows Authentication for this exercise.

Setup should complete. If you get an error here.... you might need to open a case with Microsoft... as some hotfixes can possibly block additional OpsMgr roles from being added, such as the web console. I have 951380, 954049, and 956240 installed. I was not able to add the web console.... due to the following error:

Error 1334.The file File196.2FD07918_9082_437D_99BC_FD43602A4625 cannot be installed because the file cannot be found in cabinet file Data.Cab. This could indicate a network error, an error reading from the CD-ROM, or a problem with this package.
MSI (s) (00:84) [12:38:44:863]: Product: System Center Operations Manager 2007 -- Error 1334.The file File196.2FD07918_9082_437D_99BC_FD43602A4625 cannot be installed because the file cannot be found in cabinet file Data.Cab. This could indicate a network error, an error reading from the CD-ROM, or a problem with this package.

If you are affected by this.... (common also when hotfixes wont apply) we need to do a little work in the registry.... open up HKCR\Installer\Products\DF6E5EFF035E66C49971553D96AA0E4D\Patches

Back this key up by exporting it first.... once backed up... delete the REG_SZ GUIDS, and then open the "Patches" REG_MULTI_SZ key, and delete all guids from there. When done - it should look like so:

****Note: If you are running a OpsMgr management group that was originally installed as RTM, then upgraded to SP1 - you might need to leave the following guids in place in the registry when attempting to use this workaround:

727B3A3ADCF2D1945BFF1FD34105570A (this references MOM2007QFEPreSP1.msp)
8CABA70B215243145A51419A9073262F (this references MOM2007SP1.msp)

OR - I have seen these on x64:

727B3A3ADCF2D1945BFF1FD34105570A is MOM2007QFEPreSP1.msp
8817A55B3D84652468BCF9B1E587B78F is MOM2007SP1.msp

Now - rerun setup....

Ok.... When setup is complete.... one thing we need to discuss. KB 954049 is required for Server 2008 support. If you had already applied this hotfix, you must now re-apply it in order to patch the web console files in the hotfix. The simplest way is to find the MSP file for your OS version in the C:\Program Files\System Center 2007 Hotfix Utility\ folders.

And, once installed... make sure you re-import your original reg backup we took. This workaround will typically get you through a web console add, or a hotfix install.

Once that is covered - lets test the console, from the management server itself. Launch the web console from the shortcut on the start menu.

What you will likely see... is one or more security prompts asking for your username and password.... the console it trying to use Windows Auth. Once this fails, you will be presented with a forms based authentication screen.... or an error.

If you check the OpsMgr event log - you will likely see these errors:

Log Name:      Operations Manager
Source:        Web Console
Date:          9/24/2008 1:06:11 PM
Event ID:      10
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      OMMS3.opsmgr.net
Description:
Instance: 5ogbhfrszo2xqx45iw2wid45.

Error: Data Abstraction Layer: Exception while connecting to the server 'omrms.opsmgr.net'
Thread was being aborted.

This means we need to set up Kerberos constrained delegation, so that Windows Auth can work.

1. Check the SPN of the domain account used for the SDK service account. For instance... my domain is OPSMGR, my SDK Account is OPSMGR\momsdk07, and my RMS is OMRMS.opsmgr.net. I will begin... by inspecting the SPN's attached to my SDK account:

setspn /L OPSMGR\momsdk07

Results:

Registered ServicePrincipalNames for CN=momsdk07,OU=SCOM,OU=Accounts,OU=US,DC=opsmgr,DC=net:
MSOMSdkSvc/OMRMS
MSOMSdkSvc/omrms.opsmgr.net

This is good. If for any reason these are missing - we need to add the MSOMSdkSvc class SPN of the RMS computer, to the domain account used for the SDK. So in my case, this would look like:

setspn /a MSOMSdkSvc/OMRMS OPSMGR\momsdk07

setspn /a MSOMSdkSvc/OMRMSopsmgr.net OPSMGR\momsdk07

2. Verify Domain Functional Level: If you are configuring constrained delegation, you need to verify that the domain controller is operating at Windows Server 2003 functional level. (Note: This is required for constrained delegation.) Launch "Active Directory Domains and Trusts" with domain admin credentials. In the console tree, right-click the domain for which you want to verify the domain level select Properties in the context menu.

3. Verify user account options.

Open AD Users and Computers, and find the SDK account. Examine the properties, account tab, and ensure that "Account is sensitive and cannot be delegated" is NOT selected.

4. Configure constrained delegation:

In ADUC, find the computer account that the web console is installed on.

Right click it, choose properties, and select the Delegation tab.

If in a Windows Server 2003 domain, on the Delegation tab, click Trust this computer for delegation to specified services only.

And choose the Use Kerberos only radio button.

Click the Add button

In the Add Services dialogue click the Users and Computers button

In the Select Users or Computers dialogue specify the domain account that the SDK service is running under and click OK.

In the Add Services dialogue select the service type MSOMSdkSvc and click OK.

Click OK to close the Properties Dialogue. When complete - it will appear as:

Once this is complete - Constraint Delegation is set up. You might need to wait for AD replication, and might need to bounce the SDK service on the RMS for this to start working.

These constrained delegation steps work perfectly for Windows Server 2003 - however you might not be successful in Server 2008. For my Server 2008 Web Console, I had to change the Delegation option for the Web Console server, to "Trust this Computer for delegation to any service (Kerberos only)"....

What hotfixes should I apply ?

kevinhol — Sat, 13 Sep 2008 00:01:00 GMT

This topic has moved here:

http://blogs.technet.com/kevinholman/archive/2009/01/27/which-hotfixes-should-i-apply.aspx

How do I know if I have all the required hotfixes on Windows Server 2008 to be supported for a OpsMgr agent?

kevinhol — Tue, 29 Jul 2008 02:58:27 GMT

Given the recent support of Server 2008 with an OpsMgr 2007 agent.... (see Announcing OpsMgr support for Windows Server 2008)

We need a good way to determine if all of our Server 2008 machines have the required hot-fixes.

We can do this by creating a Extended Class, and adding attributes to it from WMI queries. Sounds more complicated than it is. Let's get started:

In the authoring pane, create a new attribute. The name of each attribute will be the KB article number we are looking for.... and this will show up as a column in a new state view. I'll use KB951116 as the first example:

The discovery type will be "WMI Query". For the Target - we will pick "Windows Server 2008 Computer". Since this is a Class from a sealed MP - it will replace the target with the same name and add "_Extended". Lets go ahead and Name this extension with a little more detail.... so I am going to add "_QFE" to it.

Lastly - pick a Management Pack to place this extended attribute in... and name it according to your Company management pack naming standard. (you DO have a management pack naming standard - don't you???)

Namespace: root\cimv2

Query: select * from win32_quickfixengineering where HotFixID = 'KB951116'

Property Name (case sensitive): HotFixID

Frequency: 86400 seconds (1 once per day)

Click Finish.

Now - repeat the process.... for Each KB you want to see -

Give the next attribute a new name of the next KB number.

Target the new extended class this time (don't build a new extended class each time or you wont be able to see all KB's in a single view)

Modify the query for the new KB number.

When you are done - you can create a new State View - and scope it to your new Extended class

Now click the display tab - and only add the columns that make sense for this view - like the "Name" and then the KB columns:

It makes a simple view:

Now we can create a group... to show us ANY servers missing ANY required hotfixes.... and which hotfixes they need.... by targeting our new Extended class - and inserting an OR clause for all the KB's. Any missing KB's will be blank for the attribute property - so we use a "Does Not Equal" clause in this example:

Then right click the group - and show group members......

Now - we can add a new State view - and scope it to this new group:

Voila! We have a view that will only show us Server 2008 machines - and which hotfix they are missing... because it will be blank in the list.

The following pic - is missing a fake KB111111 I made up to test this:

If you desire more re-active methods.... you could write a script to alert you when a server was missing a hotfix... or even cooler - use a WMI performance Monitor to compare the queried value to what it should be... then you can have a state view and alerting based on these monitors.

Announcing OpsMgr support for Windows Server 2008

kevinhol — Mon, 28 Jul 2008 16:40:00 GMT

It's here!

First - read the overall KB article:

http://support.microsoft.com/kb/953141

You will need to download several Windows Server 2008 hotfixes, and then the OpsMgr hotfixes:

http://support.microsoft.com/kb/954049

The new supported configurations has been updated:

http://technet.microsoft.com/en-us/library/bb309428.aspx

Make sure you check out the "known issues" section to understand some of the known problems left with Server 2008 OpsMgr roles.... there will like be future hotfixes coming out to address these.

The catalog has the first batch of Server 2008 Management packs.... which include the Base OS:

http://www.microsoft.com/technet/prodtechnol/scp/opsmgr07.aspx?SCPProdID=3

MOMTEAM blog post on the new MP's:

http://blogs.technet.com/momteam/archive/2008/07/28/updated-management-packs-have-been-released-with-server-2008-support.aspx

REMEMBER - before importing ANY Management Pack - READ THE GUIDE that ships with them - there may be critical things you need to do first. When a new Native MP is replacing an older conversion MP.... you often MUST DELETE the previous old MP FIRST..... and this should be documented in the "Before importing the Management Pack" section of the guides.

A report to show all agents missing a specific hotfix

kevinhol — Sat, 28 Jun 2008 01:11:00 GMT

This is a continuation of my previous post on determining which agents are missing a hot-fix:

How do I know which hotfixes have been applied to which agents-

I wrote up a report that allows you to paste in a KB article number into the report as a parameter, and then it will show all agents that are potentially missing that hotfix. This will help you easily find agent which need to be patched and got missed for some reason.

You can run this report if you create the SQL reporting data source as specified in my previous post:

Creating a new data source for reporting against the Operational Database

Once imported - it will show up in the console. Open the report, and paste in any KB article number for a OpsMgr hotfix you have applied. The number MUST begin and end with "%".... such as %951380% as shown:

The report is attached below:

A little tidbit on Hot-fixes for OpsMgr

kevinhol — Wed, 25 Jun 2008 21:13:00 GMT

When you apply a hot-fix to a RMS, or Management Server, or Gateway server... a couple things will happen. First... it will update the server itself with whatever the hot-fix is supposed to fix... registry, DLL's, database updates, etc. Next, if the update needs to flow down to all agents... it will place a MSP file in the \AgentManagement directory under the OpsMgr installation directory.

Then, it will put the agents that report to the hot-fixed management server, into pending actions for the update. It will only place the agents reporting to that MS/RMS into pending... not all agents. For this reason - you really should patch ALL your RMS, MS, and GW's first, before approving any agents.

Then, when you "approve" an agent for the update... what it does is actually reinstall the agent, from its management server, then apply any update MSP's that are present, and that are not already installed.

So - when you apply a hot-fix to a management group - before approving any agents, it is a good idea to check your \AgentManagement directories on all MS/GW roles, and make sure the \x86 and \AMD64 folders have consistent AND CORRECT patch files present.

When you "approve" agents for the update... or perform a "repair", we recommend only doing 200 agents at a time, max. Phase the updates out in batches.

Then, use the "Patch List" view described in my previous blog post, to ensure all agents got updated. For agents that still need to be updated, simply run a "Repair" on those from the console, or patch them manually.

Any new agents that get pushed will automatically get the current hot-fixes applied, as long as the hot-fix MSP's are present in the \AgentManagent directory. However, manually installed agents must be hot-fixed manually.

Lastly... on the current batch of hot-fixes.... 950853 and 951380 BOTH update the SAME file.... mommodules.dll 950853 (memory leak) updates this file to 6.0.6278.11, and 951380 (cluster discovery) updates the same file to 6.0.6278.20. IF you are planning on applying both of these fixes... technically, you only need the latter, since it includes the previous fix.

Update 10-15-2008

Now - if you are applying 954903.... this contains mommodules.dll 6.0.6278.36 which supercedes BOTH 951380 and 950853.... so if you need all three hotfixes - just apply 954903. However - note in the picture below, if you apply two hotfixes that update the same file, the management server \AgentManagement directory still keeps the older one.... apparently the hotfix process does not understand that they update the same file, nor does it clean out the older 951380. The problem with this - is any major agent deployment will get impacted... because we will add to the install time, and impact the network worse. In this example - an agent push will be copying over the agent MSI (9MB) plus each hotfix in this directory.... while we dont have any direct guidance on this area - I would recommend removing the older hotfixes that no long apply, or are superceded by other hotfixes already in this directory.

How do I know which hotfixes have been applied to which agents?

kevinhol — Wed, 25 Jun 2008 01:42:00 GMT

***UPDATE*** A new hotfix has been released, which is a simple updated management pack.... which fixes the Patchlist table to include all hotfixes, and cleans up the formatting. I recommend you get it and install it on your SP1 environments.

http://support.microsoft.com/kb/958253

-------------------------------------------------------------------------------------

As more hot-fixes are applied to our OpsMgr 2007 SP1 environments.... how can we know which hot-fixes have been applied to our agents? How can we detect an agent that needs patching but got missed?

In MOM 2005... this was rather simple... in the Admin console, under Agent-managed Computers, there was a column called "version" which incremented the agent version number in most cases.

In OpsMgr... we do not update this field in the Administration tab. See graphic: The version here shows the major version number... like RTM 6.0.6500, SP1 6.0.6278.... etc....

So.... how do we examine this now for minor updates?

Create a new State view. Call it "Custom - Agent Patch List" or something you like. Target "Health Service" for "Show Data Related To". You can filter it further to the "Agent Managed Computer Group".

Then - personalize this view, and show the columns for "Name" and "Patch List" See graphic:

Now.... the "Patch List" column isn't super user friendly - because of the amount of text in the single column.... but it will let you see what has been installed. For instance - here is an example of KB950853 installed:

To make this a bit easier.... I wrote the following SQL query which does essentially the same thing.... you can create a web based SQL report from this and the data will be much easier to manage in Excel:

select bme.path AS 'Agent Name', hs.patchlist AS 'Patch List' from MT_HealthService hs
inner join BaseManagedEntity bme on hs.BaseManagedEntityId = bme.BaseManagedEntityId
order by path

If you want to query for all agents missing a specific hot-fix... you could run a query like this.... just change the KB number below (thanks to Brad Turner for providing the idea):

select bme.path AS 'Agent Name', hs.patchlist AS 'Patch List' from MT_HealthService hs
inner join BaseManagedEntity bme on hs.BaseManagedEntityId = bme.BaseManagedEntityId
where hs.patchlist not like '%951380%'
order by path

I have noticed, however, that this field, "Patch List" is limited to 255 characters in the database.... which I imagine will run out of space fairly soon. I will also be interested to see how we handle this table column, once SP2 comes out.... as any pre-SP2 applied hotfixes will no longer apply.

The Patch List information is discovered and updated once per day across all agents in the management group.

For a report which shows you the same information, but lets you query for all agent missing a specific hotfix - check out my more recent post with the report download:

http://blogs.technet.com/kevinholman/archive/2008/06/27/a-report-to-show-all-agents-missing-a-specific-hotfix.aspx