Kevin Holman's System Center Blog

Posts in this blog are provided "AS IS" with no warranties, and confers no rights. Use of included script samples are subject to the terms specified in the Terms of UseAre you interested in having a dedicated engineer that will be your Mic

OpsMgr 2007 R2 CU7 rollup hotfix – and my experience installing it

OpsMgr 2007 R2 CU7 rollup hotfix – and my experience installing it

  • Comments 7
  • Likes

 

This Cumulative Update (CU7) shipped back in January of 2013, but I never wrote up a quick guide on deploying it.  I am doing that now just for the purposes of those customers who want to apply CU7 to their SCOM 2007R2 environment before the upgrade to SCOM 2012.

The KB article describing the fixes, changes, and instructions:

http://support.microsoft.com/kb/2783850/

Get it from the download Center:

http://www.microsoft.com/en-us/download/details.aspx?id=36379

List of all OpsMgr R2 Cumulative Updates:

http://support.microsoft.com/kb/2453149

Here are the high level fixes:

Cumulative Update 7 for Operations Manager 2007 R2 resolves the following issues:

    • Console performs poorly when User Roles are used to scope object access.
    • Availability data is not shown for the current day when daily aggregation is used.
    • Behavior is inconsistent between some views on web console and console.
    • Log files do not roll over when the Unicode log file is monitored.
    • Several security issues are fixed.

Cumulative Update 7 for Operations Manager 2007 R2 resolves the following cross-platform issues:

    • Logical disk performance statistics are not collected for some volume types on Solaris computers.
    • Some Network Adapters on HP-UX computers may not be discovered.
    • Network adapter performance statistics are not collected for HP-UX network adapters.
    • The Solaris 8 and 9 agent may not restart after an ungraceful shutdown.

Note You can download the management packs for new cross plat operating system versions by going to the following Microsoft website:

System Center Operations Manager 2007 R2 Cross Platform Monitoring Management Packs

 

Let’s Roll:

So – first – I download it. The hotfix is about 1000MB.

Now – before your heart rate starts rising…. understand… this update combines the Cross Plat CU with the OpsMgr CU. (CU3 through CU6 did this as well) Aligning these is a very good thing – but it ends up increasing the size of the initial download. No worries though – I will demonstrate how to only have to copy specific files to lessen the impact of distributing this update to all your management servers and gateways, if copying a 1GB file around is a problem for you. Read about that here: http://blogs.technet.com/b/kevinholman/archive/2010/10/12/command-line-and-software-distribution-patching-scenarios-for-applying-an-opsmgr-cumulative-update.aspx

Next step – READ the documentation… understand all the steps required, and formulate the plan.

I build my deployment plan based on the release notes in the KB article. My high level plan looks something like this:

  1. Backup the Operations and Warehouse databases, and all unsealed MP’s.
  2. Apply the hotfix to the RMS
  3. Run the SQL script(s) update against the OpsDB AND Warehouse DB.
  4. Import the updated management packs provided.
  5. Apply the hotfix to all secondary Management Servers.
  6. Apply the hotfix to my Gateway Servers.
  7. Apply the hotfix to my agents by approving them from pending
  8. Apply the hotfix my dedicated consoles (Terminal servers, desktop machines, etc…)
  9. Apply the hotfix to my Web Console server
  10. Update manually installed agents…. well, manually.
  11. Apply the hotfix to my Audit collection servers

Ok – looks like 11 easy steps. This order is not set in stone – it is a recommendation based on logical order, from the release notes. For instance – if you wanted to update ALL your infrastructure before touching any agent updates – that probably makes more sense and would be fine.  Perhaps you have already deployed SCOM 2012 agents, then you might just be updating the infrastructure only.

****Requirement – as a required practice for a major update/hotfix, you should log on to your OpsMgr role servers using a domain user account that meets the following requirements:

  • OpsMgr administrator role
  • Member of the Local Administrators group on all OpsMgr role servers (RMS, MS, GW, Reporting)
  • SA (SysAdmin) privileges on the SQL server instances hosting the Operations DB and the Warehouse DB.

These rights (especially the user account having SA priv on the DB instances) are often overlooked. These are the same rights required to install OpsMgr, and must be granted to apply major hotfixes and upgrades (like RTM>SP1, SP1>R2, etc…) Most of the time the issue I run into is that the OpsMgr admin logs on with his account which is an OpsMgr Administrator role on the OpsMgr servers, but his DBA’s do not allow him to have SA priv over the DB instances. This must be granted temporarily to his user account while performing the updates, then can be removed, just like for the initial installation of OpsMgr as documented HERE. At NO time do your service accounts for MSAA or SDK need SA (SysAdmin) priv to the DB instances…. unless you decide to log in as those accounts to perform an update (which I do not recommend).

Ok, Lets get started.

1. Backups. I run a fresh backup on my OpsDB and Warehouse DB’s – just in case something goes really wrong. Since I haven’t grabbed my RMS encryption key in a long while – I go ahead and make a backup of that too, just to make sure I have it somewhere.

I also will take a backup of all my unsealed MP’s. You can do the backup in PowerShell, here is an example which will backup all unsealed MP’s to a folder C:\mpbackup:

Get-ManagementPack | where {$_.Sealed -eq $false} | export-managementpack -path C:\MPBackup

We need to do this just in case we require restoring the environment for any reason.

2. Apply the hotfix to the RMS.

Tip #1: Here is a tip that I have seen increase the success rate: Reboot your RMS/RMS nodes before starting the update. This will free up any locked processes or WMI processes that are no longer working, and reduce the chances of a timeout for a service stopping, file getting updated, etc.

Tip #2: If you are running any SDK based connectors – it is a good idea to stop these first. Things like a Remedy product connector service, Alert Update Connector, Exchange Correlation Engine, etc… This will keep them from throwing errors like crazy when setup bounces the SDK service.

Tip #3: If you are low on disk space, and you have previously installed prior R2-CU’s, you can uninstall those and make sure they are removed from \Program Files (x86)\System Center 2007 R2 Hotfix Utility\ directory. This can free up a substantial amount of disk space, and once applied these files are no longer necessary.

Tip #4: If you are running the Exchange Correlation Service for the Exchange 2010 MP, it might be a good idea to disable this service during the CU update. This service uses a lot of resources and would be best to keep it out of the picture for the CU process.

****Note: If applying this update to a RMS cluster – FIRST see: How to apply a SCOM hotfix to a clustered RMS

****Note: Many people struggle with OpsMgr hotfixes – for failing to follow instructions. When applying an OpsMgr hotfix – you need to copy the downloaded MSI file (such as SystemCenterOperationsManager2007-R2CU7-KB2783850-X86-X64-IA64-ENU.MSI) to EACH and EVERY Management server and Gateway. You need to INSTALL this hotfix installer utility to EACH Management Server and Gateway. Don’t try and just copy the update MSP files. This wont work and you will fail to update some components. Common complaints are that the agents never go into pending actions, or the agent update files never get copied over to the \AgentManagement folders. In almost ALL cases, people were taking a shortcut and making assumptions - Don’t.   Do copy the 1GB file to each machine, then install the hotfix utility, then run the hotfix from the splash screen that comes up (this is a bootstrapper program), immediately after installing the downloaded MSI. The only acceptable alternative to this process – is to install/extract the 1GB MSI to a workstation, and then build a command line based package as described below. For memory limited test environments – the command line method is the way to go.

Since my RMS is running Server 2008 R2 – I need to open an elevated command prompt to install any SCOM hotfixes. That is just how it is. So I launch that – and call the MSI I downloaded (SystemCenterOperationsManager2007-R2CU7-KB2783850-X86-X64-IA64-ENU.MSI). This will install the Hotfix Utility to the default location. I always recommend installing this hotfix utility to the default location. You can always uninstall the utility later to clean up disk space.

Tip: (This part may take a LONG TIME to complete if calling the 1GB file on a system will limited memory resources. This is because it must consume 1GB of RAM to open the file, temporarily. For production systems meeting the minimum supported 4GB, this probably wont be as much of an issue. For virtualized labs and test environments where you are running very limited memory, (1-2GB RAM) you will see this process take a considerable amount of time. On my 1GB memory virtualized management servers, it would not install at all. I upped them to 2GB and they took about 10-20 minutes to open and run the setup program. See section at the end of this article **Command line install** for ideas on how to mitigate this issue if affected)

Eventually – a splash screen comes up:

image

I choose Run Server Update, and rock and roll. You MUST execute the update from this “Run Server Update” UI. NO OTHER METHOD will work.

It runs through with success, I click finish – then another setup kicks off. This is by design. There should be three actual setups running consecutively (once for the core update, one for the localization, and one for Xplat.)

You could see this potentially three times:

image

Then wait around 30 seconds for any post install processes to complete, and then click “Exit” on the splash screen.

image

If you have trouble at with this stage – get some error messages, or if the installation rolls back – see the troubleshooting and known issues at the KB article and below in this post.

If you are patching a clustered RMS – you can continue the process using the link posted above – and complete the second node.

Now – it is time to validate the update applied correctly. I can see the following files got updated on the RMS in the standard install path: \Program Files\System Center Operations Manager 2007\

image

image

**note – this isn't all the files included in the hotfix package, just a spot check to make sure they are getting updated.

Next I check my \AgentManagement folder. This is the folder that any agents will get updates from. I check the \x86, \AMD64, and \ia64 directories:

image

 

3. Time to run the SQL scripts. There are 2 scripts, located on the RMS, in the CC:\Program Files (x86)\System Center 2007 R2 Hotfix Utility\KB2783850\SQLUpdate folder:

  • CU_Database.sql
  • CU_DataWarehouse.sql

Let’s start with CU_Database.sql

I simply need to open this file with SQL management studio – or edit it with notepad – copy the contents – and paste it in a query window that is connected to my Operations (OperationsManager) Database. I paste the contents of the file in my query window, it takes about a minute to complete in my lab. It will return a list of rows updated.

Next up – we now need to connect to the Warehouse database instance, and open a new query window against the OperationsManagerDW database. We will execute CU_DataWarehouse.sql which will return “Command(s) completed successfully”.

DO NOT skip step number 3 above, and do not continue on until this is completed.

4. Next up – import the MP updates. That's easy enough. They are located at C:\Program Files (x86)\System Center 2007 R2 Hotfix Utility\KB2626076\ManagementPacks\ and are named:

  • Microsoft.SystemCenter.DataWarehouse.Report.Library
  • Microsoft.SystemCenter.WebApplication.Library.mp
  • Microsoft.SystemCenter.WSManagement.Library.mp

These will upgrade existing MP’s in your environment. They take a few minutes each to import.

At this point – if you are using cross platform monitoring for Unix agents – you would upgrade the Xplat MP’s via a separate download. See the KB article for steps on this, and potentially upgrading your Unix agents if required.

System Center Operations Manager 2007 R2 Cross Platform Monitoring Management Packs

This download site contains the latest MP’s which were updated/included for CU7.

5.  Time to apply the hotfix to my management servers. I have 1 secondary MS server which is Windows Server 2008 R2 SP1. So I open an elevated command prompt to run the hotfix utility MSI,

Again – I MUST RUN SystemCenterOperationsManager2007-R2CU7-KB2783850-X86-X64-IA64-ENU.MSI on each Management server. This installs the hotfix utility, which will then launch the splash screen.

Tip: (This part may take a LONG TIME to complete if calling the 1GB file on a system will limited memory resources. This is because it must consume 1GB of RAM to open the file, temporarily. For production systems meeting the minimum supported 4GB, this probably wont be much of an issue. For virtualized labs and test environments where you are running very limited memory, you will see this process take a considerable amount of time. On my 1GB memory virtualized management servers, it would not install. I upped them to 2GB and they took about 10-20 minutes to open and run the setup program. See section at the end of this article **Command line install** for ideas on how to mitigate this issue if affected)

Once the splash screen comes up I “Run Server Update” These all install without issue (again – three setups run consecutively). I spot check the \AgentManagement directories and the DLL versions, and all look great. REMEMBER – you can sure patch all your management servers at the same time, however, your agents WILL fail over during this time because we stop the MS HealthService during the update. Keep this in mind. It is best to update management servers one at a time, synchronously, to keep your agents from failing over to the RMS and overloading it, or causing massive Heartbeat failures because they have nowhere to report to.

6. Next up – any Gateway machines here.

I “Run Gateway Update” from the splash screen, and setup kicks off. It runs three separate installs and I see the following – 3 times:

image_thumb1_thumb

Remember to spot check your DLL versions and \AgentManagement directories. They both should be updated.

7. I check my Pending Management view in the Administration pane of the console – and sure enough – all the agents that are set to “Remotely Manageable = Yes” in the console show up here pending an agent update. I approve all my agents (generally we recommend to patch no more than 200 agents at any given time.)

After the agents update – I need to do a quick spot check to see that they are patched and good – so I use the “Patchlist” column in the HealthService state view to see that. For creating a “Patchlist” view – see LINK

image

 

8. I have a few dedicated consoles which need updating. One is a desktop machine and the other is my terminal server which multiple people use to connect to the management group. So – I kick off the installer – and just choose “Run Server Update” as well. I do a spot check of the DLL files – and see the following was updated on the terminal server:

image

I can also perform a “Help > About” in the console itself – this will now show the update version for your console:

image

9. Next up – Web Consoles. I run mine on a stand alone management server, which I have already patched with CU7. So – I will simply just go check their DLL files to ensure they got updated.

From: \Program Files\System Center Operations Manager 2007\Web Console\bin

image

Additionally – there are some manual steps needed to secure the Web Console from a client side script vulnerability, per the KB Article  (you might have already done this in a previous CU):

Update the Web.Config file on the Web Console server role computers

  • To ensure that all cookies created by the web console cannot be accessed by client cscript, add the following configuration to the Web.Config file on each Web console server:

<httpCookies httpOnlyCookies="true"/>

  • If the web console is configured to run under SSL, add the following configuration to ensure all cookies are encrypted:

<httpCookies httpOnlyCookies="true" requireSSL="true"/>

Now – ONE of these lines need to be added to your web.config file. Scroll down in that file until you see the <system.web> tag. You can add one of these on a new line IMMEDIATELY after the <system.web> line. Here is mine – before and after:

image_thumb8 image_thumb9

Use the correct line based on your SSL configuration status for your web console. Reboot your web console server to pick up these changes.

10. At this point – I update ACS components on any ACS running Management servers that have already been patched with CU7 – but this time run the update and choose to “Run ACS Server Update”

image

After you update your collector…. you must run a SQL script that is included in the update.  This script will be run against your ACS database.  See the KB article for instructions.

11. Manually installed agents. I have a fair bit of these… so I will do this manually, or set up a SCCM package to deploy them. Most of the time you will have manually installed agents on servers behind firewalls, or when you use AD integration for agent assignment, or when you installed manually on DC’s, or as a troubleshooting step.

Additional Activities:

12. Since this particular environment I am updating is going from CU6 to CU7 – I need to import the latest cross platform management packs. If I am not using and don’t desire to use OpsMgr to monitor cross platform OS’s like SUSE, RedHat, and Solaris… then I can skip this step. However, if I do want to be fully up to date for Xplat monitoring – I need to ensure I have the latest Xplat MP’s available. The ones that are version .304 are current: http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=18891

13. I need to update the ACS reports, if I am using ACS. We have included in the CU6, some new reports which fix some reported issues with the reports. These can be found at:

C:\Program Files (x86)\System Center 2007 R2 Hotfix Utility\KB2783850\ACS\Reports

image

You might have already updated these in a previous CU.  They started shipping in CU5.

At this point I would browse to my Sql Reporting Services website that hosts my ACS reports, and import these RDL’s over the existing reports, or place them in a new folder for testing, and then move them later.

Now – the update is complete.

image

The next step is to implement your test plan steps. You should build a test plan for any time you make a change to your OpsMgr environment. This might include scanning the event logs on the RMS and all MS for critical and warning events… looking for anything new, or serious. Testing reporting is working, check the database for any unreasonable growth, run queries to see if anything looks bad from a most common alerts, events, perf, state perspective. Run a perfmon – and ensure your baselines are steady – and nothing is different on the database, or RMS. If you utilize any product connectors – make sure they are functioning.

The implementation of a solid test plan is very important to change management. Please don't overlook this step.

*** Command line install option

In some situations, you might want to perform a command line installation of the update on your RMS/management server. Most of the time – I don’t recommend this, because you generally need the feedback if each part was successful or not. However, there are situations where it is required.

One example is for users who have issues with the 1GB MSI file, and getting the hotfix installer running, especially on limited memory systems. For those, you can use a command line options which removes the issue.

For additional command line options, including how to make a CU package smaller, and how to patch consoles, agents, etc…. see the KB article which contains some guidance, and the following post which contains command line package ideas from a previous CU:

http://blogs.technet.com/b/kevinholman/archive/2010/10/12/command-line-and-software-distribution-patching-scenarios-for-applying-an-opsmgr-cumulative-update.aspx

Known issues/Troubleshooting:

1. New management packs cannot be edited in the authoring console after the Cumulative Update is installed
When a new management pack is created after CU4, CU5, CU6, or CU7 is installed and then an attempt is made to edit the management pack in the Authoring console, the Authoring console cannot edit the management pack because it cannot find the latest version of the Microsoft.SystemCenter.Library Management Pack (build .61 for CU4 and build .81 for CU5 and later). This is resolved – please see: http://support.microsoft.com/kb/2590414

2. CU fails to apply. The SDK or config service may not start after this, and CU fails on subsequent retries. The installation rolls back and you get a dialog box that the setup was interrupted before completion. There are two possible issues, with workarounds to this. One is caused by a general timeout, the other is a .NET 2.0 Issue due to a CRL response delay. Start with workaround “#1” and if that fails, try workaround “#2”. #2 is a fairly rare condition.

Workaround #1:

The services are timing out while trying to start. Using http://support.microsoft.com/kb/922918 set the ServicesPipeTimeout entry for all services to have 3 minutes (180000 milliseconds) and REBOOT the server. Then try and apply CU4. It should apply. You likely will see a few warning messages about failure to start the OMCFG service – just click ok and the setup will continue.

Workaround #2:

Using Follow the steps that are outlined in Microsoft Knowledge Base article KB936707

***Note: This hotfix likely will not be required. The hotfix is ONLY required if you are still running .NET 2.0 RTM. This hotfix is included in .NET 2.0SP1 and later. The hotfix does not resolve the issue, simply put – the hotfix (or .NET 2.0SP1 or later) simply ENABLES the use of a new tag in XML which will allow for disabling of CRL checking. If your RMS is on Windows Server 2008 or 2008R2 – you already have this hotfix included.

***Note: Once you have verified you have .NET 2.0 SP1 or later installed – you MUST perform the second step – which involves editing 2 application.exe.config files. The KB article is misleading in that it tells you to add this information as an entire section – which is incorrect – you must find the <runtime> section in your existing config files – and add a SINGLE new line to that existing section.

The manifest files are located on the RMS at the \Program Files\System Center Operations Manager 2007\ root directory. The manifest files will need to be edited for the config and sdk service on affected RMS. The file names are:

  • Microsoft.Mom.Sdk.ServiceHost.exe.config
  • Microsoft.Mom.ConfigServiceHost.exe.config

In between the EXISTING <runtime> and </runtime> lines – you need to ADD a NEW LINE with the following:

<generatePublisherEvidence enabled="false"/>

This solution disables CRL checking for the specified execute-ables, permanently.

3. Agent patchlist information incomplete, or CU patching failure. The agent Patchlist is showing parts of CU7, CU6, or CU5 but also CU4, CU3, CU2 or CU1 or nothing. The CU7 localization ENU update is not showing in patchlist. This appears to be related to the agents needing a reboot required by Windows Installer from a previous installation package. Once they are rebooted, and a repair initiated, the patchlist column looks correct with the CU7 and CU7 ENU (localized) information. The correct and complete patchlist information will appear as below:

System Center Operations Manager 2007 R2 Cumulative Update 7 - ENU Components; System Center Operations Manager 2007 R2 Cumulative Update 7 - KB2783850;

If you apply Cumulative Update 3 or 4 for Operations Manager 2007 R2, the pushed agent may not display the update list correctly. This issue occurs because the agent updates in Cumulative Update 3/4 for Operations Manager 2007 R2 may require a restart operation and then a repair operation. If you do not restart these servers after you apply Cumulative Update 3/4 for Operations Manager 2007 R2, the agent updates in Cumulative Update 6 for Operations Manager 2007 R2 are not applied. However, the restart required state is set on these computers. Therefore, you have to restart these computers and then repair the agent to apply the updates in Cumulative Update 6 for Operations Manager 2007 R2.

Comments
  • Thx Kevin!!

  • Thanks Kevin, I read your post "How to apply a SCOM hotfix to a clustered RMS" this was indented to CU1. Is this same for CU7 as well? I got to know from one of the engineer that dependency on SDK is removed from CU4 and later updates, the upgrade process can now be done like any other cluster based application upgrade. i.e. Passive node first-->Failover the active node to updated Node ->Then update passive node. Is this true? Any comments or suggestions are appreciated.

  • Thanks Kevin!

    A little question: What command (command line) do I need to update my ACS Server? Do I need to run both the Server and ACS update? Or just the ACS update?

    Best regards,
    Coen

  • Not sure where the disconnect is. There is a management server update that is required for management servers. Then, IF the server is also an ACS collector, there is another update that must be applied - as shown above in my article and referenced in the KB:

    After you update the ACS collector role, then you need to run a SQL script, from the KB:

    Manual operations that must be performed after you update the ACS collector

    Run the SQL scripts
    This update contains fixes that must be manually applied. These fixes are applied by running the %SystemDrive%\Windows\System32\Security\AdtServer\DbUpgV6toV7.sql file against the ACS database (OperationsManagerAC2). To do this, follow these steps:1.Log on to the computer that hosts the ACS database by using a user account that has database system administrator (SA) rights to the instance of the ACS database. To perform the database update remotely, log on to a computer that hosts SQL Server Management Studio by using a user account that has the appropriate SA rights to the ACS database.
    2.Run SQL Server Management Studio.
    3.In the Connect to Server dialog box, connect to the instance of SQL Server that hosts the ACS database. The default database name is OperationsManagerAC2.
    4.On the toolbar, click New Query.
    5.From the SQL Editor toolbar, use the Available databases option to select the ACS database.
    6.On the File menu, click Open, select the %SystemDrive%\Windows\System32\Security\AdtServer\DbUpgV6toV7.sql file that was installed by the Windows installer (.msi file), and then click Open.
    7.When the file is loaded, click Execute in the SQL Editor toolbar.
    8.View the Messages pane to check whether the Transact-SQL commands ran successfully.
    9.Exit SQL Server Management Studio.

  • Hi Kevin,

    Thank you for your answer. This helps. Do you know which commandline command I need to use to update the ACS components? Your artcile mentions all other options

    Best regards,
    Coen

  • Thanks for this wonderful blog Kevin!! Just one question, I want to upgrade directly from CU4 to CU7, I believe it will be fine i.e. I can skip CU5 & CU6 and directly go for CU7.

  • @Anish -
    Yes, you can skip those, because cumulative updates, are, well, cumulative. :-)

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
Search Blogs