Jonathan's Manage and Automate

VMM SCO SMA WAP - Solutions and Guidance

 

 Manage and Automate

                                           VMM SCO SMA WAP - Solutions and Guidance

  • Intermittent Connectivity Issues and Cluster Refresh Times with Windows 2008 R2 and SCVMM 2008 R2

    star angle

    It’s not often I get excited about Hotfixes… ok, that may not be entirely true, but I am very excited about these two. Why? They address WMI issues.

    ‘Why, Jonathan, would I care about WMI? That’s one of those obscure process things that I’ve never even seen…’, you say.

    ‘You’re right!’ I say, over-enthusiastically. ‘But WMI is the basis for most of the actions SCVMM takes on Hosts, whether it be refreshes or migrations.’

    ‘Wow! That *is* exciting!’, you say… mockingly.

    You get the point… Direct link to the updates at bottom. Read on for why this matters, and when it matters. (Hint: Always!)

    I came across these updates when working a customer case that involved SCVMM Hosts with SCOM (OpsMgr) Agents installed as well. Bottom line is the WMI service was taking a pounding and crashing occasionally. No WMI, no communication with SCVMM, and SCOM throws alerts. The WMI service will restart itself, but in the meantime anything depending on it is out of luck.

    WMI or WinRM?

    You can’t discuss SCVMM without mentioning WinRM at least once… Prior to hotfixes being installed, WinRM settings that are known to ease intermittent connectivity issues were implemented. While there is nothing inherently wrong with making these changes, install the hotfixes below first and see if the issue is resolved first. The less registry tweaking the better.

    Specifically, we saw Errors and hex codes in refresh jobs. The details may state WinRM is experiencing issues, but it is only reporting back results on the last action it performed. WMI errors bubble up through WinRM reporting often, so WMI was the real source in some instances.

    WinRM Issues

    Either of the below messages indicate there is likely a WinRM issue to address. In this case, follow the steps to update the registry on all Hosts. Make sure you install the updates further down as well!

    • No more threads can be created in the system (0x800700A4)

      Error (2912)
      An internal error has occurred trying to contact an agent on the server.contoso.com server.
      (No more threads can be created in the system (0x800700A4))

      Recommended Action
      Ensure the agent is installed and running. Ensure the WS-Management service is installed and running, then restart the agent.

    • The WS-Management service cannot process the request. The maximum number of concurrent operations for this user has been exceeded. Close existing operations for this user, or raise the quota for this user (0x803381A6)

    WinRM Resolution Steps

    MaxConcurrentOperationsPerUser (Increase threads)

    In Windows Server 2008 R2, the MaxConcurrentOperations for WinRM was deprecated.  However it was replaced with MaxConcurrentOperationsPerUser, which we can modify. Let also raise the value for MaxConcurrentOperationsPerUser using the following command. The default is '200' and we will double that to '400'.

    Open an elevated command prompt on each Host to execute this and all commands below.

        winrm set winrm/config/Service @{MaxConcurrentOperationsPerUser="400"}

    Start with the above value. This can be increased over the next week or so if the issue reoccurs. Once this value has been set, you’ll need to stop/restart WinRM and the VMMAgent.

        net stop winrm
        net start winrm
        net start vmmagent

    WinRM Timeout
    Use the following command to increase the default value for WinRM timeout.

        winrm set winrm/config @{MaxTimeoutms = "1800000"}

    Start with the above value, and then we can increase it over the next week or so if the issue reoccurs.  Once this value has been set, you’ll need to stop/restart WinRM and the VMMAgent.

        net stop winrm
        net start winrm
        net start vmmagent

    Reference
        Installation and Configuration for Windows Remote Management
        http://msdn.microsoft.com/en-us/library/aa384372(VS.85).aspx

    WinRM or WMI Issues

    Install all previous updates and the two new ones below as indicated for your environment.

    Error (2916)
    VMM is unable to complete the request. The connection to the agent server.contoso.com was lost.
    (Unknown error (0x80338012))

    Recommended Action
    Ensure that the WS-Management service and the agent are installed and running and that a firewall is not blocking HTTP traffic. If the error persists, reboot server.contoso.com and then try the operation again.


    Error (2927)
    A Hardware Management error has occurred trying to contact server server.contoso.com.
    (Unknown error (0x803381b9))

    Recommended Action
    Check that WinRM is installed and running on server server.contoso.com. For more information use the command "winrm helpmsg hresult".

    What Do the Hotfixes Address?

    1. This hotfix specifically addresses this issue (found in the Application log), along with general memory leaks.

    The "Win32_Service" WMI class leaks memory in Windows Server 2008 R2 and in Windows 7
    http://support.microsoft.com/kb/981314

    Example Application log error:

    Faulting application name: wmiprvse.exe, version: 6.1.7600.16385

    Symptoms

    Wmiprvse.exe crashing randomly on Windows 2008 R2 server.

    This continues even after increasing memory quota for wmiprvse.exe

    Event:

    Source:        Application Error

    Event ID:      1000

    Description:

    Faulting application name: wmiprvse.exe, version: 6.1.7600.16385, time stamp: 0x4a5bc794

    2. This hotfix addresses SCVMM Failover Cluster refresh times, among other things.

    An application or service that queries information about a failover cluster by using the WMI provider may experience low performance or a time-out exception
    http://support.microsoft.com/?id=974930

    Get your updates!

    • Install on all Windows 2008 R2 systems managed by SCVMM 2008 R2

    The "Win32_Service" WMI class leaks memory in Windows Server 2008 R2 and in Windows 7
    http://support.microsoft.com/kb/981314

    • Install on all Failover Cluster Nodes

    An application or service that queries information about a failover cluster by using the WMI provider may experience low performance or a time-out exception
    http://support.microsoft.com/?id=974930

  • SCVMM 2008 Error (2927) A Hardware Management error has occurred trying to contact server

    Toolbox tools repair fix Red

    A colleague of mine has posted a great article on one cause of these errors. Turns out there are specific refreshers that can be targeted providing WinRM and SCVMM the extra time needed to complete the job. Be sure to read the entire article carefully as there are implications with upgrading to SCVMM 2008 R2. Great article Robert!

     

    Error in SCVMM

    Error (2927)
    A Hardware Management error has occurred trying to contact server <your host name comes here>.

    (Unknown error (0x80338029))

    As it turned out, during refresh we did not wait long enough for a wsman response. The operation that timed out was to enumerate the SAN LUNs.

    In SCVMM 2008 this timeout is 120 seconds by default. Fortunately, we can increase this timeout to a maximum of 300 seconds.

    Read the entire article:

    SCVMM 2008 Error (2927) A Hardware Management error has occurred trying to contact server
    http://blogs.msdn.com/robertvi/archive/2009/10/23/scvmm-2008-error-2927-a-hardware-management-error-has-occurred-trying-to-contact-server.aspx

     

    ...

  • Computer stops responding or restarts during Hyper-V Live Migration on Windows 2008 R2

    exclamation point sign red triangle

    Hyper-V Issue that can affect SCVMM operations

    What: A new article has been released with a hotfix to resolve an issue where Windows 2008 R2 servers running on AMD Family 10h processors may restart unexpectedly or stop responding. No error messages are generated.

    When: Issue may occur when the Live Migration feature is used to move a virtual machine (VM) from one Windows Server 2008 R2 Hyper-V based node to another. This implies that the nodes are members of a Failover Cluster. This problem usually affects the source node that is the original location of the VM.

    Why: Erratum 383 in AMD Family processors generates a machine check exception.

    Hotfix Information: To apply the hotfix the computer must be running Windows Server 2008 R2. You may have to restart the computer after installing the hotfix.

    SCVMM Note: Install this hotfix on all Windows Server 2008 R2 systems that are nodes of a Failover Cluster.

    Microsoft Support URL

    The computer stops responding or restarts during the Hyper-V Live Migration process in Windows Server 2008 R2
    http://support.microsoft.com/default.aspx?scid=kb;EN-US;981618

  • SCVMM reports error 10637 when Red Hat Linux selected as the Operating System

    Wrench tools

    A new KB article regarding Red Hat is being posted on Microsoft Support any day now. For those of you who do not need the official KB, here’s the content. This is a relatively minor fix. If you are not a SQL expert, find one, or just follow the simple instructions for the free SQL Studio Express Edition. Thanks!

    URL when published http://support.microsoft.com/kb/2022557.

    Symptoms

    On System Center Virtual Machine Manager 2008 R2, it is possible to select Red Hat as the virtual machine 'Operating System' property for a virtual machine hosted by Hyper-V. The job, however, will show as failed with Error 10637. This can occur when creating a new virtual machine or changing this property on an existing VM. This does not affect the function or stability of the virtual machine itself. This property is found on the General tab of the virtual machine properties dialog.

    Error (10637)

    The virtualization software on host server.contoso.com does not support the Red Hat Enterprise Linux 5 operating system. 

    Recommended Action

    Specify a host with different virtualization software and then try the operation again.

    Cause

    A database entry that identifies the list of supported virtual machine operating systems per hypervisor requires an update. The following actions will remain unavailable for Red Hat virtual machines in SCVMM following this update:

    • Shut down of the guest operating system from the SCVMM Administrator Console
    • Detection and display of the VM Additions version for a virtual machine in the SCVMM Administrator Console and with Windows PowerShell
    • Display of the computer name of the guest operating system
    Resolution

    Using SQL Server Management Studio Express, or a full SQL installation, make a full backup of the SCVMM database. Next, connect to the SCVMM database (VirtualManagerDB by default). Open a new query and enter the commands below. The number of affected columns will be returned in the 'Messages' window if present.

    update tbl_IL_OS

    set OSFlags=0x14

    where Name like 'Red Hat Enterprise Linux 5%'

  • How to correct issue where ‘Operating System’ for Windows 2000 servers does not appear properly

    This is a recently discovered issue that has already been posted by Michael Michael on his great blog site, and there is an upcoming KB article as well. I’ve assembled step by step instructions for the SQL-challenged such as myself. The SQL tools required are free to download and explained at the bottom of this post. Enjoy!

    multi tool pocket knife toolProblem

    Windows 2000 Server and Advanced Server virtual machines display an Operating System property of 'Unknown.' If the Operating System property is set manually, it will revert to 'Unknown' following an automatic SCVMM refresh. There are no errors associated with this change.

    Resolution

    Follow Michael’s post here, or the step by step instructions below. Same content, more pictures here. ;)

    Instructions

    1. Close the SCVMM Admin Console

    2. Open the MICROSOFT$VMM$ database

    clip_image001[4]

    3. Open the VirtualManagerDB database. Your database has a different name if you changed this during SCVMM install.

    clip_image002[4]

    4. Click ‘New Query’ at the top of the window. Enter the query below as two lines and click ‘Execute’ (or press F5).

    INSERT INTO tbl_IL_OS (OSId, Name, Description, Edition, ProductType, Version, Architecture, OSFlags, VMWareGuestId)

    VALUES ('08f954f9-6475-4e07-9e32-4d2ddefc4c54', 'Windows 2000 Advanced Server', 'Windows 2000 Advanced Server', 1, 3, '5.0', 'x86', 0x3f, 'win2000AdvServGuest')

    INSERT INTO tbl_IL_OS (OSId, Name, Description, Edition, ProductType, Version, Architecture, OSFlags, VMWareGuestId)

    VALUES ('e85f1375-c69e-4cbd-8249-0e32caa04abb', 'Windows 2000 Server', 'Windows 2000 Server', 0, 3, '5.0', 'x86' , 0x3f, 'win2000ServGuest')

    5. You should see the results below under ‘Messages’.

    (1 row(s) affected)

    (1 row(s) affected)

    6. Close the SQL Management Studio, answering No or Cancel to any prompts.

    7. Restart the VMMAgent and VMMService services.

    8. Open the Admin Console and refresh your Hosts. Within 30 minutes any Windows 2000 VMs that have Integration Components (Virtual Guest Services) installed will show the correct Operating System property.

    SQL Tools

    Use SQL Server Studio Management Express (or SQL) 2005 or 2008. Microsoft SQL Server Management Studio Express can be downloaded free here:

    http://www.microsoft.com/downloads/details.aspx?FamilyID=08e52ac2-1d62-45f6-9a4a-4b76a8564a2b&displaylang=en

    Prerequisites – The SQL Express installer will prompt to install these for you

    Note: You must have administrative rights on the computer to install Microsoft SQL Server 2008 Management Studio Express. We recommended that you read the Release Notes and Readme before installing Microsoft SQL Server 2008 Management Studio Express.

    Step 1: Download and install Microsoft .Net Framework 3.5 SP1.
    Step 2 Download and install Windows Installer 4.5.
    Step 3 Download and install Windows PowerShell 1.0.

     

  • SCVMM Comprehensive Updates

    emergency light There are many updates available for SCVMM and the technologies it manages. Staying up to date can be difficult. There is a tool, the VMMCA, that will check the configuration and updates installed on the SCVMM Server and any Hosts specified and produce a simple report. This covers a great number of required updates. Now, how about all the updates that have been released since the VMMCA was last released? This article lists all of the updates the VMMCA already checks for as well as those that you will have to verify on your own. I’ll note that there is another post on this blog that explains how to check for updates across many computers automatically, given that firewall settings are as they should be. That post can be found here.

    Updates the VMMCA already verifies

    Install on the SCVMM Server
    961983
    971244

    Install on SCVMM Hosts
    950050
    956589
    956774
    958124
    954563
    955805

    Hyper-V updates for SCVMM Hosts
    952247
    956697
    957967
    958184
    959978
    971677

    Failover Cluster Management updates for SCVMM Hosts
    951308
    958065

    New updates, not verified by the VMMCA

    I do not have these broken down by installation location as in the above list. I’ll try to get this sorted out. In the meantime, here are the technologies affected and what OS and SCVMM versions they apply to. Mike is the other guru locating these updates. Be sure to visit his site as well. We use these with our support customers on a daily basis to resolve issues. If it will install on your system, install it!

    WinRM-Win2003
    936059    An update is available for the Windows Remote Management feature in Windows Server 2003 and in Windows XP

    WMI - Win2008, SP2   
    968936    Memory corruption may occur with the Windows Management Instrumentation (WMI) service on a computer that is running Windows Server 2008 or Windows Vista Service Pack 1

    WMI - Win2008, SP2   
    971403    A rollup hotfix package for Windows Server 2008 Failover Clustering WMI provider

    WMI - Win2008, SP2   
    970520    The Wmiprvse.exe process creates a memory leak on a computer that is running Windows Server 2008 if you remotely monitor this process by using the WMI interface on a computer that is running Windows Server 2003 or Windows XP

    WMI - Win2008R2   
    974930    An application or service that queries information about a failover cluster by using the WMI provider may experience low performance or a time-out exception

    WMI - Win2008R2   
    981314    The "Win32_Service" WMI class leaks memory in Windows Server 2008 R2 and in Windows 7

    Hyper-V - Win2008R2   
    981618    The computer stops responding or restarts during the Hyper-V Live Migration process in Windows Server 2008 R2

    P2V - Win2000   
    834010    A deadlock occurs when a program that uses WMI calls the LoadLibrary() or the FreeLibrary() function in Windows 2000

    P2V - Win2000, 2003   
    892294    A WMI event notification query does not detect a user permissions change on Windows 2000 or Windows Server 2003

    P2V - Win2000   
    843527    The Win32_SCSIController WMI class cannot obtain SCSI controller information after you install the MS04-011 security update

    P2V - SCVMM2008   
    959596    Description of the System Center Virtual Machine Manager 2008 update to address physical to virtual (P2V) issues

    P2V - SCVMM2008   
    971816    Using P2V together with System Center Virtual Machine Manager 2008 may fail with error 3154 (0x8099319E) or error 13252 (0x809933C4)

    VMM Rollup - SCVMM 2008R2   
    976244    Description of the System Center Virtual Machine Manager 2008 R2 hotfix rollup package: November 10, 2009

    VMM Rollup - SCVMM 2008R2   
    978560    Description of the System Center Virtual Machine Manager 2008 R2 hotfix rollup package: February 9, 2010

    Other - SCVMM2008R2   
    976246    When you remove a virtual hard disk from a virtual machine in System Center Virtual Machine Manager 2008 R2, the .vhd file on the Hyper-V server is deleted without warning

    Conclusion

    This article is clearly a work in progress, and will be updated as regularly as needed. If you are experiencing any issues with SCVMM, or simply want to prevent them… install these updates. They matter. Thanks!

     

     

    ..

April, 2010

April, 2010