Kevin Holman's System Center Blog

Posts in this blog are provided "AS IS" with no warranties, and confers no rights. Use of included script samples are subject to the terms specified in the Terms of UseAre you interested in having a dedicated engineer that will be your Mic

UR3 for SCOM 2012 R2 – Step by Step

UR3 for SCOM 2012 R2 – Step by Step

  • Comments 29
  • Likes

 

 

KB Article for OpsMgr:  http://support.microsoft.com/kb/2965445

KB Article for all System Center components:  http://support.microsoft.com/kb/2965090

Download catalog site:  http://catalog.update.microsoft.com/v7/site/Search.aspx?q=2965445

 

Key fixes:

 

  • Reliability fix:  A deadlock condition occurs when a database is connected after an outage. You may experience this issue may when one or more HealthServices services in the environment are listed as Unavailable after a database goes offline and then comes back online.  Management servers cannot reconnect to SQL after a SQL outage because of thread exhaustion. 
  • The Desktop console crashes after exception TargetInvocationException occurs when the TilesContainer is updated. You may experience this issue after you leave the console open on a Dashboard view for a long time.
  • The Password expiration monitor is fixed for logged events. To make troubleshooting easier, this fix adds more detail to Event IDs 7019 and 7020 when they occur.
  • The Health service bounces because of high memory usage in the instance MonitoringHost: leak MOMModules!CMOMClusterResource::InitializeInstance. This issue may be seen as high memory usage if you examine monitoringhost.exe in Performance Monitor. Or, the Health service may restart every couple of days , depending on the load on the server.
  • The Health service crashes in Windows HTTP Services (WinHTTP) if the RunAs account is not read correctly.
  • Windows PowerShell stops working with System.Management.Automation.PSSnapInReader.ReadEnginePSSnapIns. You may see this issue as Event ID 22400 together with a description of "Failed to run the Powershell script."
  • The PropertyValue column in the contextual details widget is unreadable in smaller widget sizes because the PropertyName column uses too much space.
  • The update threshold for monitor "Health Service Handle Count Threshold" is reset to 30,000. You can see this issue in the environment, and the Health Service Handle Count Threshold monitor is listed in the critical state.
  • An acknowledgement (ACK) is delayed by write collisions in MS queue when lots of data is sent from 1,000 agents.
  • The execution of the Export-SCOMEffectiveMonitoringConfiguration cmdlet fails with the error "Subquery returned more than 1 value.”
  • The MOMScriptAPI.ReturnItems method can be slow because a process race condition may occur when many items are returned, and the method may take two seconds between items. Scripts may run slowly in the System Center Operations Manager environment.
  • When you are in the console and click Authoring, click Management Pack, click Objects, and then click Attributes to perform a Find operation, the Find operations seems unexpectedly slow. Additionally, the Momcache.mdb file grows very large.
  • A delta synchronization times out on SQL operations with Event ID 29181.
  • Operations Manager grooms out the alert history before an alert is closed.
  • The time-zone settings are not added to a subscription when non-English display languages are set. Additionally, time stamps on alert notifications are inaccurate for the time zone.
  • Web Browser widget requires the protocol (http or https) to be included in the URL.
  • You cannot access MonitoringHost's TemporaryStoragePath within the PowerShell Module.
  • The TopNEntitiesByPerfGet stored procedure may cause an Operations Manager dashboard performance issue. This issue may occur when a dashboard is run together with multiple widgets. Additionally, you may receive the following error message after a time-out occurs:

[Error] :DataProviderCommandMethod.Invoke{dataprovidercommandmethod_cs370}( 000000000371AA78 )
An unknown exception was caught during invocation and will be re-wrapped in a DataAccessException. System.TimeoutException: The operation has timed out.  at Microsoft.EnterpriseManagement.Monitoring.DataProviders.RetryCommandExecutionStrategy.Invoke(IDataProviderCommandMethodInvoker invoker) at Microsoft.EnterpriseManagement.Presentation.DataAccess.DataProviderCommandMethod.Invoke(CoreDataGateway gateWay, DataCommand command)

 
Xplat updates:
  • Slow results are returned when you run the Get-SCXAgent cmdlet or view UNIX/Linux computers in the administration pane for lots of managed UNIX/Linux computers.
    Note To apply this hotfix, you must have version 7.5.1025.0 or later of the UNIX/Linux Process Monitoring, UNIX/Linux Log File Monitoring, and UNIX/Linux Shell Command Template management pack bundles.
  • Accessing the UNIX/Linux computers view in the administration pane can sometimes trigger the following exception message:

    Microsoft.SystemCenter.CrossPlatform.ClientLibrary.Common.SDKAbstraction.ManagedObjectNotFoundException

 

Lets get started.

From reading the KB article – the order of operations is:

  1. Install the update rollup package on the following server infrastructure:
    • Management servers
    • Gateway servers
    • Web console server role computers
    • Operations console role computers
  2. Apply SQL scripts.
  3. Manually import the management packs.
  4. Update Agents

Now, we need to add another step – if we are using Xplat monitoring – need to update the Linux/Unix MP’s and agents.

       5.  Update Unix/Linux MP’s and Agents.

 

 

1.  Management Servers

image

Since there is no RMS anymore, it doesn’t matter which management server I start with.  There is no need to begin with whomever holds the RMSe role.  I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.

I can apply this update manually via the MSP files, or I can use Windows Update.  I have 3 management servers, so I will demonstrate both.  I will do the first management server manually.  This management server holds 3 roles, and each must be patched:  Management Server, Web Console, and Console.

The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location:

image

Then extract the contents:

image

Once I have the MSP files, I am ready to start applying the update to each server by role.

***Note:  You MUST log on to each server role as a Local Administrator, SCOM Admin, AND your account must also have System Administrator (SA) role to the database instances that host your OpsMgr databases.

My first server is a management server, and the web console, and has the OpsMgr console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt:

image

This launches a quick UI which applies the update.  It will bounce the SCOM services as well.  The update does not provide any feedback that it had success or failure.  You can check the application log for the MsiInstaller events for that:

Log Name:      Application
Source:        MsiInstaller
Date:          8/6/2014 3:00:46 PM
Event ID:      1022
Task Category: None
Level:         Information
Keywords:      Classic
User:          OPSMGR\kevinhol
Computer:      SCOM01.opsmgr.net
Description:
Product: System Center Operations Manager 2012 Server - Update 'System Center 2012 R2 Operations Manager UR3 Update Patch' installed successfully.

You can also spot check a couple DLL files for the file version attribute. 

image

Next up – run the Web Console update:

image

This runs much faster.   A quick file spot check:

image

Lastly – install the console update (make sure your console is closed):

image

A quick file spot check:

image

 

 

Secondary Management Servers:

image

I now move on to my secondary management servers, applying the server update, then the console update. 

On this next management server, I will use the example of Windows Update as opposed to manually installing the MSP files.  I check online, and make sure that I have configured Windows Update to give me updates for additional products:

image29

This shows me two applicable updates for this server:

image

I apply these updates (along with some additional Windows Server Updates I was missing, and reboot each management server, until all management servers are updated.

 

Updating Gateways:

image

I can use Windows Update or manual installation.

image

The update launches a UI and quickly finishes.

Then I will spot check the DLL’s:

image

I can also spot-check the \AgentManagement folder, and make sure my agent update files are dropped here correctly:

image

 

 

2. Apply the SQL Scripts

In the path on your management servers, where you installed/extracted the update, there are two SQL script files: 

%SystemDrive%\Program Files\System Center 2012\Operations Manager\Server\SQL Script for Update Rollups

image

First – let’s run the script to update the OperationsManager database.  Open a SQL management studio query window, connect it to your Operations Manager database, and then open the script file.  Make sure it is pointing to your OperationsManager database, then execute the script.

image

Click the “Execute” button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.

You will see the following (or similar) output:

image47

or

image

IF YOU GET AN ERROR – STOP!  Do not continue.  Try re-running the script several times until it completes without errors.  In a large environment, you might have to run this several times, or even potentially shut down the services on your management servers, to break their connection to the databases, to get a successful run.

Technical tidbit:  This script has been updated in UR3.  Even if you previously ran this script in UR1 or UR2, you must run this again.

 

image

Next, we have a script in UR3 to run against the warehouse DB.  Do not skip this step under any circumstances.    From:

%SystemDrive%\Program Files\System Center 2012\Operations Manager\Server\SQL Script for Update Rollups

Open a SQL management studio query window, connect it to your OperationsManagerDW database, and then open the script file UR_Datawarehouse.sql.  Make sure it is pointing to your OperationsManagerDW database, then execute the script.

If you see a warning about line endings, choose Yes to continue.

image

Click the “Execute” button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.

You will see the following (or similar) output:

image

 

 

3. Manually import the management packs?

image

We have 6 updated MP’s to import  (MAYBE!).

image

The TFS MP bundles are only used for specific scenarios, such as DevOps scenarios where you have integrated APM with TFS, etc.  If you are not currently using these MP’s, there is no need to import or update them.  I’d skip this MP import unless you already have these MP’s present in your environment.

The Advisor MP’s are only needed if you are using System Center Advisor services.

However, the Image and Visualization libraries deal with Dashboard updates, and these need to be updated.

I import all of these without issue.

image

 

 

4.  Update Agents

image

Agents should be placed into pending actions by this update (mine worked great) for any agent that was not manually installed (remotely manageable = yes):

 image

If your agents are not placed into pending management – this is generally caused by not running the update from an elevated command prompt, or having manually installed agents which will not be placed into pending

You can approve these – which will result in a success message once complete:

image

 

Soon you should start to see PatchList getting filled in from the Agents By Version view under Operations Manager monitoring folder in the console:

image

 

 

5.  Update Unix/Linux MPs and Agents

image

Next up – I download and extract the updated Linux MP’s for SCOM 2012 SP1 UR3

http://www.microsoft.com/en-us/download/details.aspx?id=29696

7.5.1025.0 is current at this time for SCOM 2012 R2 UR2. 

****Note – take GREAT care when downloading – that you select the correct download for R2.  You must scroll down in the list and select the MSI for 2012 R2:

image

Download the MSI and run it.  It will extract the MP’s to C:\Program Files (x86)\System Center Management Packs\System Center 2012 R2 Management Packs for Unix and Linux\

Update any MP’s you are already using.   These are mine for RHEL, SUSE, and the universal Linux libraries:

image

You will likely observe VERY high CPU utilization of your management servers and database server during and immediately following these MP imports.  Give it plenty of time to complete the process of the import and MPB deployments.

Next up – you would upgrade your agents on the Unix/Linux monitored agents.  You can now do this straight from the console:

image

image

You can input credentials or use existing RunAs accounts if those have enough rights to perform this action.

image

 

 

6.  Update the remaining deployed consoles

image

This is an important step.  I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc.  These should all get the UR3 update.

 

 

 

Review:

Now at this point, we would check the OpsMgr event logs on our management servers, check for any new or strange alerts coming in, and ensure that there are no issues after the update.

image

 

 

Known issues:

See the existing list of known issues documented in the KB article.

1.  Many people are reporting that the SQL script is failing to complete when executed.  You should attempt to run this multiple times until it completes without error.  You might need to stop the Exchange correlation engine, stop the services on the management servers, or bounce the SQL server services in order to get a successful completion in a busy management group.  The errors reported appear as below:

------------------------------------------------------
(1 row(s) affected)
(1 row(s) affected)
Msg 1205, Level 13, State 56, Line 1
Transaction (Process ID 152) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
Msg 3727, Level 16, State 0, Line 1
Could not drop constraint. See previous errors.
--------------------------------------------------------

Comments
  • The Known Issues section of the KB article indicates that the agents need to be updated via the Repair Install process, not by simply approving them from the Pending Management view. Is this not correct? Your article indicates that they can be updated normally from the Pending Management view.

  • Thank you Kevin for a detailed step-by-step! Really appreciate it.

  • @Steve -

    The KB article is in error. I will notify those who own that to get it updated. Those were issues in UR2, in my testing they do not exist as issues in UR3. Thanks for pointing that out.

  • Thank You.

  • Good to see you back! However, this webpage appears to render images with the word "image" and not a real pic in ff and in IE I get the red X.

  • We did the update here and even in our tiny environ (87 agents, 2 MS, 2 GW), we had to stop the SCOM services on our MSs to do the SQL scripts. Otherwise, it went flawlessly, we had a couple of agents that didn't update, but that had nothing to do with SCOM (full C: drives, anyone?)

  • Thank you

  • Glad you are back. Thanks for the details!

  • We update our agents manually (via SCCM). So far I have installed the URS agent update manually to about 10 servers. Every server wanted to reboot. I thought Microsoft committed to no reboots for agent updates. We can suppress the reboot in SCCM but will this cause any issues with the agent?

  • Hi Kevin,

    A few days ago I've started receiving alerts in SCOM regarding "Failed to store data in the Data Warehouse. The operation will be retried.
    Exception 'InvalidOperationException': The given value of type String from the data source cannot be converted to type nvarchar of the specified target column".

    I've searched through the net and didn't find any proper solution regarding this issue.
    In addition since I get these errors I'm experiencing problems with my RMS. Every time and now it turns to gray along with the Gateway server and only after I restart the SCOM services it returns to green.

    1. Do you think that these issues are related ?
    2. Is this fix mentioned in the post here can fix this issue ?

    Thx in advance.

    Dudu Sakharovich

  • Hi Kevin,

    I installed the Update Rollups 3 on each system / role. Now when I run the first SQL script (UR_Datawarehouse.sql) against the OperationsManagerDW Database I get the following errors:I've installed all components successfully and I followed the right order by updating the components.
    Msg 208, Level 16, State 1, Procedure GetStateDaily, Line 63
    Invalid object name 'vManagedEntityMonitor'.
    Msg 208, Level 16, State 1, Procedure GetStateHourly, Line 63
    Invalid object name 'vManagedEntityMonitor'.

  • Hi actually i'm planning to install new SCOM 2012 R2 so shall i install only UR3 or start from UR1 until UR3.
    Thanks

  • Xahmed: As far as I know, these updates are cumulative, so you don't need to deploy UR1 and UR2, you can just deploy UR3.

    Kevin: Thanks for this comprehensive guide on updating. Love the part about checking DLL versions to see if the update actually was applied.

  • cool and useful docu, thanks a lot!

  • Here is a simple work around.

    Known issues: Transaction (Process ID 152) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

    Stop all management server services (Monitoring, Data Access, Management system) on all the Management servers and rerun the SQL update.

    Results Command completed Successfully.

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
Search Blogs