Kevin Holman's System Center Blog

Posts in this blog are provided "AS IS" with no warranties, and confers no rights. Use of included script samples are subject to the terms specified in the Terms of UseAre you interested in having a dedicated engineer that will be your Mic

OpsMgr 2012: Update Rollup 2 ships, and my experience installing it

OpsMgr 2012: Update Rollup 2 ships, and my experience installing it

  • Comments 1
  • Likes

 

Originally I had not planned an update article on this, but I have been getting a lot of questions on it, so lets just throw one out there.

 

Cumulative Update Rollup 2 (UR2) for OpsMgr 2012 has shipped.  This, like other CU’s or Update Rollups, is *Cumulative*.  This means that you can apply UR2 directly to a SCOM 2012 deployment with no previous updates, OR you can apply UR2 to a SCOM 2012 UR1 level management group.

Download: http://www.microsoft.com/en-us/download/details.aspx?id=30421

KB link: http://support.microsoft.com/kb/2706783

Here is a list of the fixes:

  • The Windows PowerShell module runspace configuration cache grows indefinitely. This causes memory usage to increase.
  • The Set-SCOMLicense cmdlet fails if the management group evaluation expiration time-out has expired.
  • The System Center Operations Manager agent may crash on Oracle Solaris root zones that are configured to use a range of dedicated CPUs.
  • The UNIX/Linux agent process provider may not enumerate running processes if a process has arguments that include non-ASCII characters. This prevents process/daemon monitoring.
  • The .rpm specification file for the agent for Red Hat Enterprise Linux does not specify the distribution.

 

 

Let’s Roll:

So – first – I download it. The hotfix file “SystemCenter2012OperationsManager-UR2-KB2731874-X86-X64-IA64-ENU.exe” is ONLY about 76 megabytes

NOTE:  Cumulative Updates/Update Rollups for SCOM 2012 has changed in a very fundamental way from the old Cumulative Updates in SCOM 2007.  No longer is there a bootstrapper program for the CU for SCOM 2012. Now – the package simply extracts the files to a directory.  We will then run each MSP file independently based on whatever server roles are installed.  It will not detect your installed roles – you must handle this, and apply which updates are applicable.

 

Next step – READ the documentation… understand all the steps required, and formulate the plan.

I build my deployment plan based on the release notes in the KB article. My high level plan looks something like this:

  1. Backup the Operations and Warehouse databases, and all unsealed MP’s.
  2. Apply the hotfix to all the Management Servers
  3. Apply the hotfix to my Gateway Servers.
  4. Apply the hotfix to the OpsMgr Reporting server.
  5. Apply the hotfix to my Web Console server
  6. Apply the hotfix my dedicated consoles (Terminal servers, desktop admin machines, etc…)
  7. Import the management pack updates
  8. Agents: Apply the hotfix to my agents by approving them from pending.
  9. Agents: Update manually installed agents…. well, manually.
  10. Unix/Linux: Download and import the updated MP’s for Unix/Linux monitoring.
  11. Unix/Linux: Upgrade the Unix/Linux agent providers.

Ok – looks like 11 easy steps. This order is not set in stone – it is a recommendation based on logical order, from the release notes.

****Requirement – as a required practice for a major update/hotfix, you should log on to your OpsMgr role servers using a domain user account that meets the following requirements:

  • OpsMgr administrator role
  • Member of the Local Administrators group on all OpsMgr role servers (RMS, MS, GW, Reporting)
  • SA (SysAdmin) privileges on the SQL server instances hosting the Operations DB and the Warehouse DB.

These rights (especially the user account having SA priv on the DB instances) are often overlooked. These are the same rights required to install OpsMgr, and must be granted to apply major hotfixes and upgrades (like RTM>SP1, SP1>R2, etc…) Most of the time the issue I run into is that the OpsMgr admin logs on with his account which is an OpsMgr Administrator role on the OpsMgr servers, but his DBA’s do not allow him to have SA priv over the DB instances. This must be granted temporarily to his user account while performing the updates, then can be removed, just like for the initial installation of OpsMgr as documented HERE. At NO time do your service accounts for MSAA or SDK need SA (SysAdmin) priv to the DB instances…. unless you decide to log in as those accounts to perform an update (which I do not recommend).

Ok, Lets get started.

 

1. Backups. I run a fresh backup on my OpsDB and Warehouse DB’s – just in case something goes really wrong.

I also will take a backup of all my unsealed MP’s. You can do the backup in PowerShell, here is an example which will backup all unsealed MP’s to a folder C:\mpbackup:

Get-SCOMManagementPack | where {$_.Sealed -eq $false} | export-SCOMmanagementpack -path C:\MPBackup

We need to do this just in case we require restoring the environment for any reason.

 

2. Apply the hotfix to the Management Servers.

Pro Tip #1: Here is a tip that I have seen increase the success rate: Reboot your Management Server nodes before starting the update. This will free up any locked processes or WMI processes that are no longer working, and reduce the chances of a timeout for a service stopping, file getting updated, etc. This is not a requirement, just something to consider if you have had issues applying such a fix.

Pro Tip #2: If you are running any SDK based connectors – it is a good idea to stop these first. Things like a Remedy product connector service, Alert Update Connector, Exchange Correlation Engine, etc… This will keep them from throwing errors like crazy when setup bounces the SDK services.

I start by running the download, SystemCenter2012OperationsManager-UR2-KB2731874-X86-X64-IA64-ENU.exe. This is simply an extractor. You can run this anywhere, and provide a location for the update. No longer do we need to run this extractor on each server role. We can now simply extract these files to a network share, and update all server roles from there.

Here are the files after extraction:

image

 

I start on my first management server, OMMS1. This has the Server role, Web Console Role, and Console installed. So I will need to run 3 MSP’s.

I will start by running KB2731874-AMD64-Server.msp.  Right click the file and choose “apply” or call it from an elevated CMD.

Pro Tip: Open an ELEVATED COMMAND PROMPT and run these MSP' files from the command prompt. User Access Control (UAC) will still block a successful install in many cases, so you will experience greater success if you always run updates from an elevated CMD.  Even if you *think* you don’t have UAC enabled.

Pro tip:  The “AMD64” files are for any 64bit server.  This is what you will always execute on a server role, since SCOM 2012 ONLY supports 64 bit servers for server roles.  The “i386” files were included only for 32bit agents, and where the console is installed on 32bit workstations.

 

You will see a dialogue box like below:

 

image

image

The server update goes pretty quickly, depending on how long your server takes to restart the OpsMgr services.  It bounced the DAS, Config, and SC Mgmt services during the update. 

***Note – you MIGHT get a message that a reboot is required, if any of the files to be updated are locked and cannot be replaced.  If so, I recommend rebooting before continuing.

Next, I will update the web console by running KB2731874-AMD64-WebConsole.msp. This only takes a few seconds.

Next, I will update the Console files. I need to make sure that I close any open consoles on this server, from my session or any other logged in sessions via RDP. I will apply KB2731874-AMD64-Console.msp. Again – this only takes a few seconds to run.

Now – I am done updating all three installed server roles on this server. I can spot check to make sure the files got updated:

Server role update: From \Program Files\System Center 2012\Operations Manager\Server

image

Web Console role update: From \Program Files\System Center 2012\Operations Manager\WebConsole\WebHost\bin

image

Console role update: From \Program Files\System Center 2012\Operations Manager\Console

image

 

Next – I am ready to update the rest of my management servers. I only have one other MS – OMMS2. This only runs the console and server roles, so only two MSP’s to apply.

I apply each MSP, takes a couple minutes.

Another spot-check to make, is to each that the Agent binaries get updated on each Management Server role, for Windows agents. These are located at: \Program Files\System Center 2012\Operations Manager\Server\AgentManagement\ in the AMD64 or i386 directory:

Check to ensure that the CU dropped the appropriate agent update file, such as KB2731874-amd64-Agent.msp.

image

 

3. Apply the hotfix to my Gateway Servers.

I don’t have any gateways in my lab right now – but this would be a very simple execution of the KB2731874-AMD64-Gateway.msp file.

 

4. Apply the hotfix to the OpsMgr Reporting server.

I have SCOM reporting and SSRS installed on a SQL server, DB01.

I log on and apply KB2731874-AMD64-Reporting.msp. This runs in a few seconds.

Spot check: From \Program Files\System Center 2012\Operations Manager\Reporting\Tools

image

 

5. Apply the hotfix to my Web Console server

I already did this in step 2 – but I could have waited and done this now, or patched any dedicated Web Console servers.

 

6. Apply the hotfix my dedicated consoles (Terminal servers, desktop machines, etc…)

I need to apply this update anywhere the console is installed – including consoles installed on management servers. I have already updated the console patch on my management servers OMMS1 and OMMS2. Now I have a Terminal Server where I run the console – so I will patch this server now, which only runs the console. Make sure you close the console and close any other user sessions out – or you might require a reboot to finish the update for the console files which will be locked by an open console. This also includes any open Powershell windows connected to the SDK. You will get a warning if setup detects any. You can run the same spotcheck for the file version update as above. (Note – help/about will not show the updated version in the console – this stays the same major version, so don’t go looking there.)

 

7. Import the management pack updates

Open a console – and import the files previously extracted:

Microsoft.SystemCenter.DataWarehouse.Library.mp

Microsoft.SystemCenter.Visualization.Library.mpb

Microsoft.SystemCenter.WebApplicationSolutions.Library.mpb

image

These import in a few minutes without a hitch.

 

8. Agents: Apply the hotfix to my agents by approving them from pending

I open the console – Administration > Device Management > Pending Management, and see all my agents in pending that are not manually installed (Remotely Manageable = Yes)

Let me stop and talk about how agents get into Pending. This is a ONE TIME operation, which is created at the exact moment that you run the CU on a management server.  What will happen, is that the CU runtime will look for all agents ASSIGNED to that management server, that are Remotely Manageable (not manually installed) and will put ONLY those agents into pending at that time. We will NEVER go back and re-inspect to put old agents into pending, because this is not a SCOM workflow that handles this.  It is done only by applying the update. If you don’t have agents in pending – you either aren't running the MSP from an elevated cmd, or you aren't running the update as a user that is BOTH a SCOM admin, Local OS Admin, and SQL Systems Administrator role to the databases.

 

image

 

I right click all mine – provide an account that has rights to deploy/install the update remotely, and kick it off.

100% Success!

How can I tell?  Open the Console, Monitoring > Operations Manager > Agent Details > Agents by Version (state view):

image

9. Agents: Update manually installed agents…. well, manually.

I simply run the file KB2731874-AMD64-Agent.msp or KB2731874-i386-Agent.msp depending on which OS version (64bit or 32bit) I am updating. I can deploy these manually, or via software distribution packaging up each MSP and applying them to the correct OS by version/architecture.

10. Unix/Linux: Download and import the updated MP’s for Unix/Linux monitoring

The Unix/Linux update files are located in a separate download. This includes new versions of the management packs, and updated agent binaries. I go to the link in the KB article and grab these update files. Monitoring Pack for UNIX and Linux Operating Systems.msi. It is the typical MP installer/extractor program. Run setup and extract these files to your preferred location. I generally just recommend installing/extracting these to your default location on any workstation/server, and then copying the extracted files to your OpsMgr software/MP/Updates network share.

Download link: http://www.microsoft.com/en-us/download/details.aspx?id=29696

Next – import all the files for your versions f Unix/Linux Agents that you support. The RTM version of the Unix/Linux MP’s was 7.3.2026.0. The UR1 version is 7.3.2097.0.  The UR2 version is 7.3.2119.0.

***Note – several of these new MP files are actually in the new .MPB format. This new format allows management packs to take advantage of the new schema extensions in System Center 2012, to be able to transfer binaries among other items. Such as the agent binaries

image

 

11. Unix/Linux: Upgrade the Unix/Linux agent providers

One you have imported the updated MP files for Unix/Linux agents, spot check that these files got updated on your management servers. Look in the following folder:

\Program Files\System Center 2012\Operations Manager\Server\AgentManagement\UnixAgents\DownloadedKits

 

image

 

Notice the new update files are version 1.3.0-218. Also, it may take some time for this folder/files to show up, after importing the MP’s, as your database and management servers will be experiencing high CPU utilization from SDK and Config activities, and this is normal.  So be patient. You need to ensure that these files are updated on any members of your management server resource pool that monitors Unix/Linux agents before continuing. 

*** Note – there is a minor issue with these Unix/Linux files. They should be copied over to this folder automatically, however they are not. You might have to restart the System Center Management service, after waiting for 10-20 minutes for the post import processes to complete. After bouncing the System Center Management service on each management server, the binaries should show up. HOWEVER – it does appear we are leaving behind the previous version files, so you will see the new 218 version and the previous 214 version from UR1 if you applied that.  This should be cleaned up in a future cumulative update.

 

Now – to update my Unix/Linux agents. I can use the Update-SCXAgent cmdlet in powershell, or I can use the Administration wizard. I right click my version -214 agent, and choose Upgrade Agent.

image

 

image

You can use an existing RunAs account profile if you have configured these with an agent upgrade account that has the necessary rights, or you can input new credentials using an account that has SSH privileges in order to remotely install software on the Unix/Linux agent machine. If all goes well – you get a success:

image

image

Now – the update is complete.

image

The next step is to implement your test plan steps. You should build a test plan for any time you make a change to your OpsMgr environment. This might include scanning the event logs on the Management servers for critical and warning events… looking for anything new, or serious. Testing reporting is working, test your web console, check the database for any unreasonable growth, run queries to see if anything looks bad from a most common alerts, events, perf, state perspective. Run a perfmon – and ensure your baselines are steady – and nothing is different on the database, or management servers. If you utilize any product connectors – make sure they are functioning.

The implementation of a solid test plan is very important to change management. Please don't overlook this step.

Comments
  • Hi Kevin, just wondering. You make no mention of this section of the original guide reagarding the Web Console:

    •The web console fixes will work after you add the following line to the %windir%\Microsoft.NET\Framework64\v2.0.50727\CONFIG\web.config file:

    <machineKey validationKey="AutoGenerate,IsolateApps" decryptionKey="AutoGenerate,IsolateApps" validation="3DES" decryption="3DES"/>

    Note This line should be added under <system.web>, as described in the following article in the Microsoft Knowledge Base:

    911722

    I assume it is necessary for the upgrade, is that right?

    Thank you

    Jose

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
Search Blogs