Kevin Holman's System Center Blog

Posts in this blog are provided "AS IS" with no warranties, and confers no rights. Use of included script samples are subject to the terms specified in the Terms of UseAre you interested in having a dedicated engineer that will be your Mic

UR2 for SCOM 2012 R2 – Step by Step

UR2 for SCOM 2012 R2 – Step by Step

  • Comments 28
  • Likes

 

Sorry I am a bit behind in publishing this post.  Smile

image

 

KB Article:   http://support.microsoft.com/kb/2929891

Download catalog site:  http://catalog.update.microsoft.com/v7/site/Search.aspx?q=2929891

 

Key fixes:

Issue 1 - This update rollup makes the stored procedure performance aggregate more robust against out-of-range values.
Issue 2 - Adding multiple regular expressions (RegEx) to a group definition causes an SQL exception when the group is added or run.
Issue 3 - Web applications fail when they are monitored by the System Center Operations Manager 2012 R2 APM agent.
Issue 4 - Service Level Objectives (SLO) dashboards sometimes load in several seconds and sometimes take minutes to load. Additionally, the dashboard is empty after it loads in some cases.
Issue 5 - Operations Manager Console crashes when you try to override the scope in the Authoring pane.
Issue 6 - The System Center Operations Manager console is slow to load views if you are a member of a custom Operator role.
Issue 7 - This update rollup includes a fix for the dashboard issue that was introduced in Update Rollup 1.
Issue 8 - SQL Time Out Exceptions for State data (31552 events) occur when you create Data Warehouse workflows.
Issue 9 - This update rollup includes a fix for the Event Data source.

Xplat updates:
Issue 1 - All IBM WebSphere application servers that run on Linux or AIX computers are not automatically discovered by the Management Pack for Java Enterprise Edition (JEE) if multiple application servers are defined in a single WebSphere profile.

 

Lets get started.

From reading the KB article – the order of operations is:

 

  1. Install the update rollup package on the following server infrastructure:
    • Management servers
    • Gateway servers
    • Web console server role computers
    • Operations console role computers
  2. Apply SQL scripts (see installation information).
  3. Manually import the management packs.
  4. Update Agents

Now, we need to add another step – if we are using Xplat monitoring – need to update the Linux/Unix MP’s and agents.

       5.  Update Unix/Linux MP’s and Agents.

 

1.  Management Servers

   image

Since there is no RMS anymore, it doesn’t matter which management server I start with.  There is no need to begin with whomever holds the RMSe role.  I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.

I can apply this update manually via the MSP files, or I can use Windows Update.  I have 3 management servers, so I will demonstrate both.  I will do the first management server manually.  This management server holds 3 roles, and each must be patched:  Management Server, Web Console, and Console.

The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location:

image

 

Then extract the contents:

image

Once I have the MSP files, I am ready to start applying the update to each server by role.

***Note:  You MUST log on to each server role as a Local Administrator, SCOM Admin, AND your account must also have System Administrator (SA) role to the database instances that host your OpsMgr databases.

My first server is a management server, and the web console, and has the OpsMgr console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt:

image

This launches a quick UI which applies the update.  It will bounce the SCOM services as well.  The update does not provide any feedback that it had success or failure.  You can check the application log for the MsiInstaller events for that:

Log Name:      Application
Source:        MsiInstaller
Date:          6/2/2014 1:58:33 PM
Event ID:      1035
Task Category: None
Level:         Information
Keywords:      Classic
User:          OPSMGR\kevinhol
Computer:      SCOM01.opsmgr.net
Description:
Windows Installer reconfigured the product. Product Name: System Center Operations Manager 2012 Server. Product Version: 7.1.10226.1015. Product Language: 1033. Manufacturer: Microsoft Corporation. Reconfiguration success or error status: 0.

 

You can also spot check a couple DLL files for the file version attribute. 

image

 

Next up – run the Web Console update:

image

This runs much faster.   A quick file spot check:

image

Lastly – install the console update (make sure your console is closed):

image

A quick file spot check:

image

 

Secondary Management Servers:

image

I now move on to my secondary management servers, applying the server update, then the console update. 

On this next management server, I will use Windows Update.  I check online, and make sure that I have configured Windows Update to give me updates for additional products:

image29

This shows me two applicable updates for this server:

image

I apply these updates (along with some additional Windows Server Updates I was missing, and reboot each management server, until all management servers are updated.

 

Updating Gateways:

image

I can use Windows Update or manual installation.

image

The update launches a UI and quickly finishes.

Then I will spot check the DLL’s:

image

That said – there is a long running bug in the gateway update.  The gateway update is NOT placing a very important file here – for agents.

BUG:  In the \Program Files\System Center Operations Manager\Gateway\AgentManagement\ directories – we should be dropping an agent update MSP file for updating agents behind gateways, for x86 and amd64 agents.  However, the GW update does not include this.  If you want to push-deploy agents behind gateways, and need them to be fully up to date, you should copy the correct files from your updated management servers directories.

 

 

2. Apply the SQL Script

 

In the path on your management servers, where you installed/extracted the update, there are two SQL script files: 

%SystemDrive%\Program Files\System Center 2012\Operations Manager\Server\SQL Script for Update Rollups

image

First – let’s run the script to update the OperationsManager database.  Open a SQL management studio query window, connect it to your Operations Manager database, and then open the script file.  Make sure it is pointing to your OperationsManager database, then execute the script.

image44

Click the “Execute” button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.

You will see the following (or similar) output:

image47  

or

 image

IF YOU GET AN ERROR – STOP!  Do not continue.  Try re-running the script several times until it completes without errors.  In a large environment, you might have to run this several times, or even potentially shut down the services on your management servers, to break their connection to the databases, to get a successful run.

Technical tidbit:  If you had previously ran this script by applying it during the application of SCOM 2012 R2 UR1, this script is unchanged in UR2.  Therefore it does not have to be executed again during the UR2 deployment.  There is no harm in running it again, especially if you are not 100% sure it was run with success during the UR1 deployment, if applicable.  Always best to just run it again with the deployment of UR2.  However, if you have a large environment and it is difficult to get the script to execute with success, you might skip this step.  Again – only if you already applied UR1, and you are 100% sure it was run with success then.

 

image

Next, we have a new script in UR2 to run against the warehouse DB.  Do not skip this step under any circumstances.    From:

%SystemDrive%\Program Files\System Center 2012\Operations Manager\Server\SQL Script for Update Rollups

Open a SQL management studio query window, connect it to your OperationsManagerDW database, and then open the script file UR_Datawarehouse.sql.  Make sure it is pointing to your OperationsManagerDW database, then execute the script.

If you see a warning about line endings, choose Yes to continue.

  image

Click the “Execute” button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.

You will see the following (or similar) output:

image

 

3. Manually import the management packs?

image

We have five updated MP’s to import  (MAYBE!).

image

The TFS MP bundles are only used for specific scenarios, such as DevOps scenarios where you have integrated APM with TFS, etc.  If you are not currently using these MP’s, there is no need to import or update them.  I’d skip this MP import unless you already have these MP’s present in your environment.

The Advisor MP’s are only needed if you are using System Center Advisor services.

However, the Image and Visualization libraries deal with Dashboard updates, and these need to be updated.

I import all of these without issue.

 

 

4.  Update Agents

image

There is a known issue in UR2 for agents – read carefully below:

Agents should be placed into pending actions by this update (mine worked great):

image

If your agents are not placed into pending management – this is generally caused by not running the update from an elevated command prompt, or having manually installed agents which will not be placed into pending

You can approve these – which will result in a success message:

image

HOWEVER – this didn’t actually do any update.  You can see from the system event logs, that MOMAgentinstaller did run, but when we check the DLL versions, we can see they are not updated.

What you need to do is REJECT any pending updates in the SCOM console – then run a REPAIR on your agents to get them to apply the update.  Alternatively – use a software distribution tool like Configuration Manager to apply agent updates where applicable.  Any agents that are manually installed (Remotely Manageable = No) will not be available for a repair, as always. 

You can track running repairs in Pending Management:

image

 

Soon you should start to see PatchList getting filled in from the Agents By Version view under Operations Manager monitoring folder in the console:

image

 

 

 

5.  Update Unix/Linux MPs and Agents

image

Next up – I download and extract the updated Linux MP’s for SCOM 2012 SP1 UR2

http://www.microsoft.com/en-us/download/details.aspx?id=29696

7.5.1021.0 is current at this time for SCOM 2012 R2 UR2. 

****Note – take GREAT care when downloading – that you select the correct download for R2.  You must scroll down in the list and select the MSI for 2012 R2:

image50

 

Download the MSI and run it.  It will extract the MP’s to C:\Program Files (x86)\System Center Management Packs\System Center 2012 R2 Management Packs for Unix and Linux\

Update any MP’s you are already using.

image

You will likely observe VERY high CPU utilization of your management servers and database server during and immediately following these MP imports.  Give it plenty of time to complete the process of the import and MPB deployments.

Next up – you would upgrade your agents on the Unix/Linux monitored agents.  You can now do this straight from the console:

image

image

You can input credentials or use existing RunAs accounts if those have enough rights to perform this action.

 

 

 

5.  Update the remaining deployed consoles

image

This is an important step.  I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc.  These should all get the UR2 update.

 

 

Review:

Now at this point, we would check the OpsMgr event logs on our management servers, check for any new or strange alerts coming in, and ensure that there are no issues after the update.

image

Known issues:

See the existing list of known issues documented in the KB article.

1.  Many people are reporting that the SQL script is failing to complete when executed.  You should attempt to run this multiple times until it completes without error.  You might need to stop the Exchange correlation engine, stop the services on the management servers, or bounce the SQL server services in order to get a successful completion in a busy management group.  The errors reported appear as below:

------------------------------------------------------
(1 row(s) affected)
(1 row(s) affected)
Msg 1205, Level 13, State 56, Line 1
Transaction (Process ID 152) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
Msg 3727, Level 16, State 0, Line 1
Could not drop constraint. See previous errors.
--------------------------------------------------------

2.  Gateway Servers don’t get agent patch update files.  See body of this blog article for more details.

3.  Agents don’t go into pending, or go into pending but the agent update doesn’t actually work.  This is a known issue and will be addressed in the next UR3.  For this release, simply use a “repair” to repair the agents that need the update, or use a software distribution mechanism to deploy the update.

Comments
  • Nice to have it!! And welcome back!! Hope the new arrivals are doing well !

  • Hi Kevin, great article as always. I was following your steps in our SCOM environment, everything worked great but I have a question about the SQL scripts. I have already applied UR1 to this environment and there was a "update_rollup_mom_db.sql" script in "C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\SQL Script for Update Rollups" directory, which I executed during UR1 deployment.. After UR2 installation, that script seems not to have changed, the "last access" date property of the file show the day when I applied UR1, and the script has a creation date of "1/13/2014". The new "UR_Datawarehouse" script and the MPs in "Management Packs for Update Rollups" (also there were MPs here from UR1 and all of them seems updated) have all last access date of today.

    Is there any way to confirm if this SQL script is the UR2 updated version? If not, is there a manual way to get the script without reinstalling the update? Thanks.

  • @ Batuhan

    I noticed the same thing - the date of the script for update_rollup_mom_db.sql did not change - so it looks like there weren't any changes from UR1 to UR2. I hate to post in an article when not to run something, because a customer might think they applied it previously when they didn't.... so the safe bet is to re-run it. That said, if it was run with success during UR1 application, this specific script should not need to be re-run. I will verify with the PG and post back to make sure that assumption is correct and nothing changed.

  • @ Batuhan -

    I verified - there were no changes from UR1 for update_rollup_mom_db.sql , so IF you are 100% sure you applied this correctly during UR1 application, there is not a requirement to apply this again.

  • Kevin,
    thanks for the information provided in your article. I used it to deploy this update to our SCOM environment.

    Keep up the good work.

    Kind regards

  • So what is the newest agent version? All my agents are version 7.1.10184.0 and I don't see anything change when I repair the agents. Also don't see them show up in Pending management.

  • @Raoul - Agent version is never updated. The agent version is 2012R2. That does not change. What changes with these UR's is the "Patch List" property which is described above in the blog post.

  • Welcome back Kevin.

    This might be a stupid question but is it worth waiting for UR3. I did the update in a test environment and the upgrade just smelt bad. The warning for the line ending with the DW SQL script and agents failing to update even after indicating the process completed after the agent approval.

    I did as you mentioned and repaired the agents but am still deciding if I should wait till UR3 for my production environment. Hopefully UR3 the agent update issues are sorted and the process is a little more polished.

    This is the first UR I have done and was a bit disappointed that I had to go digging to see if the update completed successfully for both the agent and the management servers.

    Thanks,

    Mike

  • You can check your agent version/patch from Powershell:

    $agents = get-scomagent
    $value = $agents[0].HostedHealthService.GetMonitoringProperties() | ? {$_.Name -eq “PatchList”}
    $agents | select version, Name, PrimaryManagementServerName, ManuallyInstalled, @{Label=”PatchList”; Expression={$_.hostedhealthservice.getmonitoringpropertyvalue($value)}} | sort primarymanagementservername, manuallyinstalled, version | ft -autosize

    If the PatchList value is empty, then you don't have the latest UR update on the agent.

  • Thx again Kevin.

    Just updated the Prod environment (after running Windows Update and rebooting) and the agents came up as UR2 automagically (except for 3 servers).
    In Test, which I did a couple of weeks ago, they had to be updated as per the blog post.
    Go figure....
    JB

  • FYI - There is also another issue that we have run into. The issue is centered around the # of State tables that have been created to hold the aggregated data and the SQL Log file size. In our case we had 455 State Hourly tables and our log file was able to grow to 100GBs. To attack this issue we first increase the SQL log to grow without restriction. (Note: We had several TBs of free space on the server to be able to do this) After the upgrade there is an expensive SQL query that re-indexes all the State Houly and Daily tables. If you have a large number of aggregated tables the SQL Script will timeout with the default value on the OperationsManagerDW as the query is written to proceed through ALL of the tables associated with Hourly State data. I believe the default timeout is 3 hours. During this expensive query there is a schema lock on the OperationsManagerDW preventing the management servers from writing data to it. The fix is to increase the SQL Query Timeout via a registry key on each of the Management Servers and restart the services. (reg add "HKLM\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Data Warehouse" /v "Deployment Command Timeout Seconds" /t REG_DWORD /d 43200 /f) Roughly 15 minutes after the start of the Management Services, the SQL script will run. Allow it to finish. In our case it took just over 4 1/2 hours to complete and the SQL log file grew to 375GBs!! Once complete the Schema Lock lifts allowing data to be written and Dataset Maintenance Aggregation functions to complete. We experienced this on several OpsMgr Environments within our infrastructure. So I know some other people probably have run into the same issue so I thought I would share. :-)

  • Even the below single line ps works well

    get-scomagent | select Name, PatchList | Sort-Object Name

  • Thank you Mr. Kevin.

  • Hi Kevin,

    Just wondering about the additional fixes listed in KB: http://support.microsoft.com/kb/2929891:

    Enable the web console fixes
    Fix for the data warehouse BULK insert commands timeouts.

    These were not listed in your step by step. Should these only be applied in certain circumstances?

    Thanks,

  • Kevin,
    Hope all is well. Just wanted to express my gratitude for the efforts and time you dedicate toward SCOM related articles. Scom Community received many help from your work. Thank you for this. Fahim

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
Search Blogs