Welcome to TechNet Blogs Sign in | Join | Help

I see many issues that could’ve been easily resolved if the VMMCA, or Virtual Machine Manager Configuration Analyzer had been ran prior to opening up a case with Microsoft CSS. The VMMCA has now been updated for VMM R2. 

Description

The VMMCA is a diagnostic tool you can use to evaluate important configuration settings for computers that either are serving or might serve VMM roles or other VMM functions. The VMMCA scans the hardware and software configurations of the computers you specify, evaluates them against a set of predefined rules, and then provides you with error messages and warnings for any configurations that are not optimal for the VMM role or other VMM function that you have specified for the computer.

You can use VMMCA before installing VMM, to help you evaluate the configuration of computers that you might use for the following VMM roles:

  • VMM server
  • VMM Administrator Console
  • VMM Self-Service Portal

You can also use VMMCA after installing VMM, to evaluate or troubleshoot the configurations of computers that you are now using or might use for the VMM roles listed above, as well as computers that you are using or might use for any of the following VMM functions:

  • Windows Server–based host
  • VMware Virtual Center Server
  • P2V source computer
  • Operations Manager agent

System Requirements:

Before you install the VMMCA, you must download and install the 64-bit version of Microsoft Baseline Configuration Analyzer (MBCA). To download the MBCASetup64.msi file, go to http://go.microsoft.com/fwlink/?LinkID=97952.

Note    The system requirements for the MBCA indicate that only Windows Server 2003 is supported. However, the VMMCA and the MBCA have been tested and are supported on Windows Server 2008 64-bit.

You must install and run the VMMCA on the computer that either currently is or will become your VMM 2008 server. To review the system requirements for the VMM 2008 server,

go to http://go.microsoft.com/fwlink/?LinkId=133516.

 

Download VMMCA Now

Common Usage Scenarios

The list of scenarios are not comprehensive, but will provide you with some common usage scenarios to provide you with an understanding of when to use VMMCA 2008.

Scenario 1 – I have a Windows Host that is in a not responding state.  Use VMMCA 2008 to check the host for commonly known issues.

Scenario 2 – A Windows host shows a status of “Needs Attention”.  Use VMMCA 2008 to check the this host for commonly known issues.

Scenario 3 – I am using SCVMM 2008 to manage a VMware environment, but I am unable to create new virtual machines on the VMware environment. Use VMMCA 2008  to evaluate the Virtual Center environment for commonly known issues.

Scenario 4 – I am using the integration between SCVMM 2008 and Operations Manager, but I am unable to see a particular VM or Host in the integrated mapping view.  Use VMMCA 2008 to check the host or virtual machine for an operations manager agent.

Scenario 5 – I am using SCVMM 2008 to do P2V conversions and the job is failing.  Use VMMCA 2008 to check the P2V source for commonly known issues.

Scenario 6 -  I tried to create a Virtual Machine and it failed during customization and installation of virtual guest services.  Use VMMCA 2008 on the Windows Host to find commonly known issues.

 

As we all know VMM uses BITS to transfer files during various operations.  However from time to time you may find yourself needing to troubleshoot whether or not BITS is working correctly outside of VMM.  This can be performed using the following method.

BITSadmin can be downloaded here:  http://msdn.microsoft.com/en-us/library/aa362813(VS.85).aspx

BITSadmin examples here:  http://msdn.microsoft.com/en-us/library/aa362812(VS.85).aspx

 

Here are the instructions for you to test with:

 

1.     Download BITSadmin.exe using the above link. It is also in the Windows Server 2003 Resource Kit.  It does not appear in the tools directory, but is accessible from the command line.

2.     Click on … Start > Programs > Windows Resource Kit Tools > Command Shell

3.     This should open an elevated command prompt

4.     Create a C:\Temp directory if one does not already exist

5.     Type the following command: 

 

bitsadmin /transfer myDownloadJob /download /priority normal http://msdl.microsoft.com/download/symbols/debuggers/dbg_amd64_6.11.1.404.msi c:\temp\dbg_amd64_6.11.1.404.msi

 

6.     This command will automatically use BITSadmin.exe to test BITS functionality by downloading a file from our website to the local temp directory

7.     This should take 10-20 minutes to complete

 

This test should be performed anywhere you need to test BITS.  For example, if you're troubleshooting a P2V failure this should be performed on the VMM server, Host computer and Source machine.

It has come to our attention that trying to add a computer with more than 16 processors (for example a quad six-core machine with 24 processors) is crashing the VMM server with the following exception:

09435 00009433 273.19396973 [5684] 1634.16A4::07/22-20:50:29.545#26:VmRefresher.cs(182): System.InvalidOperationException: Nullable object must have a value.
09436 00009434 273.19396973 [5684] at System.Nullable`1.get_Value()
09437 00009435 273.19396973 [5684] at Microsoft.VirtualManager.Engine.Adhc.WindowsHostDataProperties.GetProcessorData()
09438 00009436 273.19396973 [5684] at Microsoft.VirtualManager.Engine.Adhc.WindowsHostDataProperties.PopulateProperties()

Here is an example of 24 processors from the SYSTEMINFO file:

System Model    IBM 3850 M2 / x3950 M2
System Type    x64-based PC
Processor    Intel(R) Xeon(R) CPU           E7450  @ 2.40GHz, 2398 Mhz, 6 Core(s), 6 Logical Processor(s)
Processor    Intel(R) Xeon(R) CPU           E7450  @ 2.40GHz, 2398 Mhz, 6 Core(s), 6 Logical Processor(s)
Processor    Intel(R) Xeon(R) CPU           E7450  @ 2.40GHz, 2398 Mhz, 6 Core(s), 6 Logical Processor(s)
Processor    Intel(R) Xeon(R) CPU           E7450  @ 2.40GHz, 2398 Mhz, 6 Core(s), 6 Logical Processor(s)

In the VM Manager log, note the following errors:

Log Name:      VM Manager
Source:           Virtual Machine Manager
Date:              7/22/2009 2:45:56 PM
Event ID:        19999
Task Category: None
Level:              Error
Keywords:        Classic
User:               N/A
Computer:       HOST.CONTOSO.com
Description:     Virtual Machine Manager (vmmservice:2516) has encountered an error and needed to exit the process. Windows generated an error report with the following parameters:
Event:VMM20
P1(appName):vmmservice
P2(appVersion):2.0.3444.0
P3(assemblyName):E.Adhc.Operations
P4(assemblyVer):2.0.3454.0
P5(methodName):M.V.E.A.WindowsHostDataProperties.GetProcessorData
P6(exceptionType):System.InvalidOperationException
P7(callstackHash):e9f1

Log Name:      VM Manager
Source:           Virtual Machine Manager
Date:              7/22/2009 2:45:56 PM
Event ID:         1
Task Category: None
Level:              Error
Keywords:       Classic
User:              N/A
Computer:      HOST.CONTOSO.com
Description:
System.InvalidOperationException: Nullable object must have a value.
   at System.Nullable`1.get_Value()
   at Microsoft.VirtualManager.Engine.Adhc.WindowsHostDataProperties.GetProcessorData()
   at Microsoft.VirtualManager.Engine.Adhc.WindowsHostDataProperties.PopulateProperties()
   at Microsoft.VirtualManager.Engine.Adhc.WindowsHostAbstractionLayer.GetHostProperties(HostData oldHostData)
   at Microsoft.VirtualManager.Engine.Adhc.HostRefresher.GatherAllInformation(Host host, Object agentRefreshSyncObj, Guid taskID)
   at Microsoft.VirtualManager.Engine.Adhc.HostRefresher.RefreshLockedHost(Host host, Guid taskID)
   at Microsoft.VirtualManager.Engine.BitBos.VMRefresherBase.UpdateHostToResponding(Nullable`1& prevState)
   at Microsoft.VirtualManager.Engine.BitBos.VMRefresherBase.UpdateHostState(IVMComputerSystemSummary[]& vmComputers, Nullable`1& prevState)
   at Microsoft.VirtualManager.Engine.BitBos.VMRefresherBase.RunLightRefresher()
   at Microsoft.VirtualManager.Engine.BitBos.VMRefresherBase.UpdateHostandVMs(VMRefresherType refresherType, Guid vmObjectId, VM tempVm)
   at Microsoft.VirtualManager.Engine.BitBos.VmLightRefresher.RefreshData(HostReference hostRef)
   at Microsoft.VirtualManager.Engine.RefreshDriver`1.RefreshThreadFunction(Object obj)-2146233079

In the 19999 error note the WindowsHostDataProperties.GetProcessorData and System.InvalidOperationException which correspond directly to the exception from the trace.  In the 1 error note the Nullable`1 also corresponds directly to the exception from the trace.

This issue is outlined on IBM’s website:  http://www-947.ibm.com/systems/support/supportsite.wss/docdisplay?brandind=5000008&lndocid=MIGR-5079202

From a Hyper-V perspective we support up to 24 cores (requires a QFE KB956710).  We have a few workarounds for VMM to work in this case (either do #1 or implement #2 and #3)

  1. You can limit the number of CPUs via msconfig to 16 (we know this solution is not ideal in this case, but wanted to document it anyway)
  2. Go to BIOS, then navigate to Advanced Setup>Clustering Technology and set it to Physical (default is Logical)
  3. Use Bcdedit to add USEPHYSICALDESTINATION and set to YES. This forces the physical APIC to be used.

 There are several publicly available hotfixes for System Center Virtual Machine Manager 2008.  Obviously they should be installed on the VMM server, but how about the Host computers?  Are they applicable there as well?  These are questions that I hear quite often so I wanted to take a few minutes and list the currently available hotfixes for VMM and discuss where they should be applied.

Most VMM related hotfixes can be found in just two articles:

·          962941 Recommended hotfixes for System Center Virtual Machine Manager 2008

·          961983 Description of the hotfix rollup package for System Center Virtual Machine Manager 2008: April 14th, 2009

The fixes outlined in 962941 need to be installed on both your VMM server and Hyper-v Host computers. 

961983 is a rollup package released on April 14, 2009 as an available update on Windows Updates. So hopefully it has already been applied to your VMM server.  However, if you're VMM server does not regularly check Windows Updates directly or for some reason does not have that particular patch installed, you can download it from here. 

To apply the path manually, run the following command line:  "msiexec /p vmmServer64update.msp BOOTSTRAPPED=1"

This is one of the most often asked questions on our support forums.  And it can also be rather difficult to troubleshoot although if you follow the steps below you can systematically approach this issue and usually resolve it.  While we're on the topic, this status indicates that the VMM server is unable to communicate with the Host for some reason.  If this communication is interrupted for any reason, even intermittently, then you can expect the Host status to change in the VMM admin console.

1.     The first item to check is to make sure that the following hotfixes are installed:

·          958124 A wmiprvse.exe process may leak memory when a WMI notification query is used heavily on a Windows Server 2008-based or Windows Vista-based computer

·          954563 Memory corruption may occur with the Windows Management Instrumentation (WMI) service on a computer that is running Windows Server 2008 or Windows Vista Service Pack 1

·          955805 Certain applications become very slow on a Windows Server 2008-based or Windows Vista S955805-based computer when a certificate with SIA extension is installed

·          961983 Description of the hotfix rollup package for System Center Virtual Machine Manager 2008: April 14th, 2009

 

2.     TCP Offloading - will need to be disabled in Windows, the registry and also in any NIC teaming management software that may be in use.  It is very important to check in all three locations to ensure that TCP Offloading is completely disabled.  This operation needs to be performed on both the VMM server and Host computer.

 

·          Locate all physical NICs in the registry under the following key:  'HKLM\System\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}'

·          There will be folders with four digits, starting with '0000'. Locate the ones that show the physical NIC names on the right. Look for the 'DriverDesc' value on the right; this should have the name of a NIC such as 'HP NC360T PCIe Gigabit Server Adapter.' For each of these, make the changes below.

            Disable All Vendor specific Offloading.  Set values for any entries below that include the word 'Offload' to '0' (Disabled)
                        *FlowControl

                                    No description available

                        *IPChecksumOffloadIPv4
                                    Describes whether the device enabled or disabled the calculation of IPv4 checksums.
                        *TCPChecksumOffloadIPv4
                                     Describes whether the device enabled or disabled the calculation of TCP Checksum over IPv4 packets.
                        *TCPChecksumOffloadIPv6
                                    Describes whether the device enabled or disabled the calculation of TCP checksum over IPv6 packets.
                        *UDPChecksumOffloadIPv4
                                    Describes whether the device enabled or disabled the calculation of UDP Checksum over IPv4 packets.
                        *UDPChecksumOffloadIPv6
                                    Describes whether the device enabled or disabled the calculation of UDP Checksum over IPv6 packets.
                        *LsoV1IPv4
                                    Describes whether the device enabled or disabled the segmentation of large TCP packets over IPv4 for large send offload version 1 (LSOv1).
                        *LsoV2IPv4
                                    Describes whether the device enabled or disabled the segmentation of large TCP packets over IPv4 for large send offload version 2 (LSOv2).
                        *LsoV2IPv6
                                    Describes whether the device enabled or disabled the segmentation of large TCP packets over IPv6 for large send offload version 2 (LSOv2).
                        *IPsecOffloadV1IPv4
                                    Describes whether the device enabled or disabled the calculation of IPsec headers over IPv4.
                        *IPsecOffloadV2
                                    Describes whether the device enabled or disabled IPsec offload version 2 (IPsecOV2). IPsecOV2 provides support for additional crypto-algorithms, IPv6, and co-existence with large send offload version 2 (LSOv2).
                        *IPsecOffloadV2IPv4
                                    Describes whether the device enabled or disabled IPsecOV2 for IPv4 only.
                        *RSS
                                    Receive side scaling
                        *TCPUDPChecksumOffloadIPv4
                                    Describes whether the device enabled or disabled the calculation of TCP or UDP checksum over IPv4.
                        *TCPUDPChecksumOffloadIPv6
                                    Describes whether the device enabled or disabled the calculation of TCP or UDP checksum over IPv6.

·          Disable Offloading in Windows.
            Use the following registry values to enable or disable task offloading for the TCP/IP protocol:  HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\TCPIP\Parameters\DisableTaskOffload

            Setting this DWORD value to ‘1’ disables all of the task offloads from the TCP/IP transport. Setting this value to zero enables all of the task offloads.

·          Disable Offloading in teaming management software.

            This is an often overlooked component of this troubleshooting process.  Many vendors have some form of Offloading capabilities built-in to their teaming management software.  This can take on many forms and is often vendor specific.

 

Additional information of Offloading can be found at the following MSDN article:

 

Using Registry Values to Enable and Disable Task Offloading

http://msdn.microsoft.com/en-us/library/aa938424.aspx

 

 

3.     WinRM and svchost.

            SCVMM is very dependent upon WinRM for its underlying communication.  So if there is a problem with the underlying WinRM communication between the VMM server and Host computer, then we can also reach this same error condition.  The clue that most often leads             me down this path is that if we reboot the Host and the status is 'Ok', but after a period of time of up to 3-4 hours the status changes back to 'Not Responding' then this may indeed be the underlying root cause. 

 

            If this is the underlying problem, then the expected behavior is that if you stop the WinRM service from a command prompt it will take much longer than normal to complete.  I've seen it take up to 5 minutes to stop in this scenario.  This issue can occur if the shared         instance of svchost that WinRM is running under becomes backed up. 

 

            As a solution, run WinRM in its own instance of svchost by typing the following command from an elevated command prompt.  The exact syntax is very important.  Notice the space after the =.  If this is successful you will see the following:

 

                        c:\>sc config winrm type= own

                        [SC] ChangeServiceConfig SUCCESS

 

 

4.     Another possible cause of this issue can be if a "Restrictive Groups" group policy is removing the VMM server machine account from the local administrators group on the host computer.  This issue is discussed in further detail in KB 969164 and is available online here.  If this is the case, move the VMM server and Host computers to a new OU that is blocking inheritance of all group policy objects.

 

5.     Some additional potential causes of this problem include:

 

·          if the VMM Agent is not running

·          if Anti-virus software is scanning ports or protocols

 

While we're on the topic of VMM security, I thought it would be helpful to list what accounts need to be where in order for VMM to function properly. 

·          VMM server machine account

            Administrators group on VMM server and all Hosts

                        Virtual Machine Manager Servers local group on the VMM server

·          The account used to perform actions in VMM

            Must be a member of the Local Administrators group on the VMM server and all Host computers

 

           

 

In summary, these steps should resolve most issues relating to the Host status of 'Not Responding'.

 

 

SecurityGood

With a single command you can pull a list of all installed updates. Use this to compare patch levels on all systems so you can easily target those not meeting the bar.

We start with simple commands, then build on these to create robust queries with optional CSV and HTML formatting. I think of these commands, which I use on a daily basis, as basics for my Toolkit. If you don’t have a Toolkit I suggest you create one now. Place your often used tools in a folder such as ‘C:\Toolkit’ so that they are always at hand. Grab the first five items under ‘Top 10 Downloads’ from Sysinternals.com to round out your basics.

speechuxcpl.dll_I03b8_0409TIP:  Go one step further and sign up for a Live Mesh account with your Windows Live ID. With this you can  keep your Toolkit directory, and other directories, synched up across all machines you designate. The folders will be available privately online as well. Further, remote control is built-in for all Mesh enabled systems. Sweet! Clearly remote capabilities are geared more towards workstations than servers, but I use it for both.Live_Mesh

And yeah, it’s free. Get started here.

 


 

Basic Commands

Warm up by opening an elevated command prompt on your SCVMM server (or any server for that matter). If your command window does not show ‘Administrator:’ as below, you are not running an elevated console.

Command_Admin

> Single command to pull updates.
wmic qfe

http://support.microsoft.com/?kbid=963660  VMM2008R2-03  Update                        KB963660
                       NT AUTHORITY\SYSTEM  2/26/2009
http://support.microsoft.com/?kbid=967062  VMM2008R2-03  Update                        KB967062
                       NT AUTHORITY\SYSTEM  2/26/2009

You should see a torrent of information scroll through the command window. This is a list of all QFEs (Hotfixes and Updates) installed, along with a good bit of extraneous info.

> Now let's work on formatting. This is my everyday method.
wmic qfe list brief

Update                        KB963660                                 NT AUTHORITY\SYSTEM  2/26/2009
Update                        KB967062                                 NT AUTHORITY\SYSTEM  2/26/2009

Now some of the noise is removed and it is easier to read the list of updates.

speechuxcpl.dll_I03b8_0409

TIP:  Is KBXXXXXX installed on this system?!? I can only remember the last three digits of the update anyway… Easy to check. Just pipe your output to ‘find.’

wmic qfe list brief | find /i "660"

Update                        KB963660                                 NT AUTHORITY\SYSTEM  2/26/2009

Apparently it is installed. Nice.

> This is good, but we can do better.
wmic qfe get csname,hotfixid,description

VMM2008R2-03  Update           KB963660
VMM2008R2-03  Update           KB967062

Now we're getting somewhere. We've selected the header for only those columns we are interested in and have returned a clean list of updates along with the system name.

This much you can keep in your head. You might want to create a batch file for items in the next section.

 


 

Advanced Queries

Now that you've waded into the deep end let's perform some real work. These are the commands to place in a batch file for your Toolkit. The steps below assume you have already created a ‘C:\Temp’ directory on your system. CSName is not required for the next two commands; it is implied.

> Create a CSV file from the output.
wmic /output:c:\temp\qfe.csv QFE GET HotFixID,Description /format:csv

The resulting CSV file lists these items, ready to be opened in Excel: Node,Description,HotFixID

> CSV files are not as interesting as HTML, so let's shift output formats.
wmic /output:c:\temp\qfe.html QFE GET HotFixID,Description /format:htable

Now you have a clean multicolored table with results, complete with column headers.


Remote Systems

The only thing left to do is perform this same work on remote machines. First a single machine, then a list of systems. Replace <remotesystem> with your remote system name. Note that we add CSName to pull the remote name.

> List remote system updates.
wmic /node:"<remotesystem>" /output:c:\temp\qfe_remote.html QFE GET CSName,HotFixID,Description /format:htable

If this did not work for you keep in mind that you must be an administrator on the remote machine, and Remote Management must be enabled.

> Finally, let's feed a list of server names and create a single HTML file. Create a file named 'serverlist.txt' with one system per line.
wmic /node:@serverlist.txt /output:c:\temp\qfe_serverlist.html QFE GET CSName,HotFixId,Description /format:htable

That's it. You should have a table similar to this.WMI_POST


Enjoy your new tools. Thanks for reading!

 

For further reference:

MSDN - WMI Command Line Tools
http://msdn.microsoft.com/en-us/library/aa394531.aspx


Content provided by Jonathan Jordan.  Jonathan is the current beta lead for VMM 2008 R2 and developed the above content.  More information from Jonathan will be coming soon, or you can check out his personal blog at http://blogs.technet.com/jonjor

"Where should I install VMM, on a physical server or inside a VM and then add that same physical server as a managed host?"  This is a common question from our customers regarding the placement of VMM. 

The answer is that this is a fully supported scenario.  In fact, some of our customers want to build a fully virtualized environment, with the only physical machines being hosts.  In this scenario, the entire environment can then be managed using one VMM server running on a virtual machine on a host that is being managed by the VMM server.

There are however some very important considerations that should be considered when planning your virtualization environment.

If the above scenario is implemented ...

1.       You will be unable to migrate the VM that contains SCVMM to another host, which will fail for obvious reasons.  As for best practices, the VM that contains SCVMM should not be migrated, turned off, paused or save state, ect.  Also, Performance Resource Optimization (PRO) should not be enabled for that host group, such that it can automatically migrate the VMM server.

2.       Another gotcha would be if the physical host went to a 'Not Responding' state.  Then you may not be able to open the admin console locally or remotely.

So while this is a fully supported scenario, it may not be the best choice for your particular environment.

The following steps outline the procedure to manually remove a host cluster from the VMM database.  In this specific example, we are removing a two node cluster.  Please note that the GUIDs listed below are specific to this example.  If following these instructions, you will need to determine this same information for your specific environment.  This information can be found in the tbl_ADHC_Host

 

1.       First, stop the VMMservice

2.       Then, take a backup of the existing DB using SQL Management studio. (Right click on DB name, choose tasks, click on Back up, chose a full backup)

3.       Now, right click on table tbl_ADHC_Host and select view or open rows

4.       Click on the SQL button from the tool bar to get the SQL query for the view

5.       In there, start pasting the commands below one by one and verifying they executed correctly. (To execute, you can highlight the command and press F5 or press the “!” symbol)

 

DELETE FROM tbl_ADHC_HostNetworkAdapter WHERE (HostID = '85099616-fdc8-4f85-a702-979a4251f050')

DELETE FROM tbl_ADHC_HostNetworkAdapter WHERE HostID='8cfff5ad-0990-4a92-bd6d-4d24190e5d5a '

DELETE FROM tbl_ADHC_VirtualNetwork WHERE (HostID = '85099616-fdc8-4f85-a702-979a4251f050')

DELETE FROM tbl_ADHC_VirtualNetwork WHERE (HostID = '8cfff5ad-0990-4a92-bd6d-4d24190e5d5a ')

DELETE FROM tbl_ADHC_HostVolume WHERE (HostID = '85099616-fdc8-4f85-a702-979a4251f050') OR (HostID = '8cfff5ad-0990-4a92-bd6d-4d24190e5d5a ')

DELETE FROM tbl_ADHC_HostDisk WHERE (HostID = '85099616-fdc8-4f85-a702-979a4251f050') OR (HostID = '8cfff5ad-0990-4a92-bd6d-4d24190e5d5a ')

DELETE FROM tbl_WLC_PhysicalObject WHERE (HostId = '85099616-fdc8-4f85-a702-979a4251f050') OR (HostId = '8cfff5ad-0990-4a92-bd6d-4d24190e5d5a ')

DELETE FROM tbl_WLC_VObject WHERE (HostId = '85099616-fdc8-4f85-a702-979a4251f050') OR  (HostId = '8cfff5ad-0990-4a92-bd6d-4d24190e5d5a ')

DELETE FROM tbl_ADHC_Host WHERE (HostID = '85099616-fdc8-4f85-a702-979a4251f050') OR  (HostID = '8cfff5ad-0990-4a92-bd6d-4d24190e5d5a ')

DELETE FROM tbl_ADHC_ClusterDisk WHERE (ClusterID = '1148853c-cde3-45d2-82ee-bb04259d6a79')

DELETE FROM tbl_ADHC_HostCluster WHERE (ClusterID = '1148853c-cde3-45d2-82ee-bb04259d6a79')

DELETE FROM tbl_ADHC_AgentServerRelation WHERE (AgentServerID = '648e977e-d10e-40b6-aca2-d9d2dc63b790')

DELETE FROM tbl_ADHC_AgentServer WHERE (ComputerName = 'Server_Node1')

DELETE FROM tbl_ADHC_AgentServer WHERE (ComputerName = 'Server_Node2')

 

       6.   Now start the vmmservice and open the VMM console.

 

 

For more information on how to manually remove virtual machines see http://blogs.technet.com/scvmm/archive/2009/04/29/how-to-forcefully-remove-a-virtual-machine-from-the-scvmm-admin-console.aspx

SUMMARY

When performing a Physical-to-Virtual (P2V) conversion to a clustered host with System Center Virtual Machine Manager 2008, it may fail with the following error:

 

Error 2927

A Hardware Management error has occurred trying to contact server %servername.

(Unknown error (0x8033801A))

 

Recommended Action

Check that WinRM is installed and running on %servername.  For more information use the command "winrm helpmsg hresult".

 

 

NOTE

The problem occurs when trying to perform a P2V to a clustered host.  If the P2V is attempted to a non-clustered host, it completes successfully.

 

 

CAUSE

This issue can occur if the clustered storage where the resulting VM from the P2V conversion is placed has an invalid character such as a "\". 

 

Example:  This is commonly seen when clustered storage is named as follows:

 

"Disk E:\" instead of "Disk E".

 

 

MORE INFORMATION

To confirm this scenario, collect a trace using Dbgview.  This process is outlined in the following article:

 

 

970066  How to collect traces in System Center Virtual Machine Manager

http://support.microsoft.com/default.aspx?scid=kb;EN-US;970066

 

 

The exception noted in the trace log will be similar to the following:

00040598             272.28906250     [3344] *** Carmine error was: HostAgentWSManError (2927); HR: 0x8033801A

 

 

To look for this problem, search for the following string in the trace:

"Select * from MSCluster_ResourceToDisk"

 

 

Here is a sample link showing an invalid character.

[Select * from MSCluster_ResourceToDisk where GroupComponent='MSCluster_Resource.Name="Disk E:\"']

 

 

RESOLUTION

Change the cluster resource name to remove the invalid character.

 

970925 SCVMM 2008 P2V conversion fails with Error 2927 (0x8033801A)
http://support.microsoft.com/default.aspx?scid=kb;EN-US;970925

Summary

In order to diagnose a failure scenario in System Center Virtual Machine Manager, it is normally required to reproduce the issue while collecting a trace using the DebugView tool.

 

 

More Information 

DebugView is an application that lets you monitor debug output on your local system, or any computer on the network that you can reach via TCP/IP. It is capable of displaying both kernel-mode and Win32 debug output, so you don't need a debugger to catch the debug output your applications or device drivers generate, nor do you need to modify your applications or drivers to use non-standard debug output APIs.

 

Here are some common definitions that will be used throughout this article:

§  VMM Server:  hub of a VMM implementation through which all other VMM components interact and communicate.  The VMM server runs the VMM service, which runs commands, transfers files, and controls communications with other VMM components and with all virtual machine hosts and VMM library servers, collectively referred to as managed computers.  The VMM server also connects to a SQL Server database that stores all VMM configuration information.

§  VMM admin consolea graphical user interface (GUI) used to manage the VMM server; can but does not have to be installed on the VMM server itself.

§  VMM Host:  physical computer that hosts one or more virtual machines.

§  Library: catalog of resources that can be used to create and configure virtual machines in VMM.

§  Source Machine: Refers to the physical machine that will be converted to a virtual machine

 

 

Important          

Traces should always be collected from all related computers. For example, if performing a P2V conversion, traces should be collected from the VMM server, the Host (if used) and the Source computer. 

 

 

Collecting a trace

1.       The first step is to install DebugView on each required machine.  DebugView can be downloaded from the following link:   http://technet.microsoft.com/en-us/sysinternals/bb896647.aspx

2.       Save the following code into a text file and name it as "odsflags.cmd":

 

@echo off

echo ODS control flags - only trace with set flags will go to ODS

 

if (%1)==() goto :HELP

if (%1)==(-?) goto :HELP

if (%1)==(/?) goto :HELP

 

echo Setting flag to %1...

reg ADD "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Tracing\Microsoft\Car

mine" /v ODSFLAGS /t REG_DWORD /d %1 /f

echo Done.

goto :EXIT

 

:HELP

echo Usage: odsflags [flag], where flag is

echo TRACE_ERROR = 0x2,

echo TRACE_DBG_NORMAL = 0x4,

echo TRACE_DBG_VERBOSE = 0x8,

echo TRACE_PERF = 0x10,

echo TRACE_TEST_INFO = 0x20,

echo TRACE_TEST_WARNING = 0x40,

echo TRACE_TEST_ERROR = 0x80,

 

:EXIT

 

3.       Save the following code into a text file and name it as "odson.reg":

 

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Tracing\Microsoft\Carmine]
"ODS"=dword:00000001

4.       Save the following code into a text file and name it as "odsoff.reg":

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Tracing\Microsoft\Carmine]
"ODS"=dword:00000000

  1. Copy the above three files onto each machine to a folder such as C:\Debgview
  2. Through Windows Explorer, navigate to this folder and double click on “odson.reg”.  This should report success.
  3. At an elevated command prompt change to this folder and run “odsflags.cmd 255”.  This should report success.
  4. Restart the following services:
    1. Restart vmmservice on VMM server with “net stop vmmservice” and “net start vmmservice”.
    2. Restart the agent service on the host with “net stop vmmagent” and “net start vmmagent”.
    3. Restart the IIS service on the Web server with "iisreset".
  5. Right click on DebugView and select “Run as administrator”.
    1. Make sure that in its Capture menu, you have both "Capture Win32" and "Capture Global Win32" checked.
    2. You should be able to see tracing from the VMM components showing up in the DebugView. (If you need to collect traces for both VMM Server and the host, make sure to do these steps on all computers.)
  6. Reproduce the issue.

11.   Once the issue has been reproduced, stop the capture on all machines. You can later save the captures, but we need the data capture to stop now.

a.       On each machine remove the check next to ‘Capture Data’ in the Capture menu

b.      On each machine then go to File> Save and save the log

12.  This log file[s] will need to be provided to Microsoft for analysis

13.   On each machine double click ‘odsoff.reg’. This should report success

 

 

References:

Additional information can be found at the following blog:

http://blogs.technet.com/chengw/archive/2008/05/08/how-to-collect-scvmm-traces.aspx

http://blogs.technet.com/mbriggs/archive/2009/02/06/how-to-capture-a-dbgview-trace-from-windows-core.aspx

 

970066 How to collect traces in System Center Virtual Machine Manager
http://support.microsoft.com/default.aspx?scid=kb;EN-US;970066

SUMMARY

When using System Center Virtual Machine Manager 2008 to perform any action involving file transfers across machines such as new VM or new P2V the operation may fail with the following error:

 

 

Error (2912)
An internal error has occurred trying to contact an agent on the %serverName%.
(Element not found (0x80070490))

 

Recommendation Action

Ensure that the agent is installed and running.  Ensure the WS-Management service is installed and running, then restart the agent.

 

0x80070490 = Element not found = Certificate not found

 

 

NOTE

The exact error message displayed in the console is generic in nature, such as 2912 or 3154. However it will always contain the "Element no found" in the description.

 

 

CAUSE

This issue is caused by a problem with the Host certificate (incorrect name, IP instead of FQDN or NetBIOS) or the certificate is missing from the VMM server.

 

 

MORE INFORMATION

Troubleshooting this issue requires capturing a trace using the Dbgview utility.  The following article discusses this process in more detail, including where to download the Dbgview utility.

 

970066  How to collect traces in System Center Virtual Machine Manager

http://support.microsoft.com/default.aspx?scid=kb;EN-US;970066

 

The following is a snippet from the stack trace while this error occurred.  Note where the operation is checking for the certificate on the VMM server.  Another key is the BitDeployer.Copy function.

 

00006921                302.15213013         [1972]    at Microsoft.VirtualManager.Engine.Deployment.FileDeploymentServerJobFactory.GetPeerCertificate(WsmanAPIWrapper clientWrapper)   

00006922                302.15213013         [1972]    at Microsoft.VirtualManager.Engine.Deployment.ImageDeploymentServerJobFactory.CreateJob(WsmanAPIWrapper serverWrapper, WsmanAPIWrapper clientWrapper, DeploymentFile file, Boolean useHTTPS)    

00006923                302.15213013         [1972]    at Microsoft.VirtualManager.Engine.Deployment.BitDeployer.Copy()  

00006924                302.15213013         [1972]    at Microsoft.VirtualManager.Engine.Deployment.DeploySubtask.RunSubtask()  

00006948                302.15328979         [1972] *** Carmine error was: HostAgentFail (2912); HR: 0x80070490            

 

 

 

RESOLUTION

The solution is to remove the managed host from the VMM server and also delete any residual certificates from the host on the VMM server, and then re-add the host.

 

 

1.       On the SCVMM server, remove the managed host from the console.  The steps on how to remove a managed host are outlined in the following TechNet article:

 

http://technet.microsoft.com/en-us/library/cc956121.aspx

 

2.       Now we need to locate and delete any certificates for the Host computer

3.       Open the Certificate console on the SCVMM server

                a.  Open a new mmc and add the certificates snap-in

                b.  Select the option of  'computer account' and 'local computer'

                c.  Select Finish and Ok to load the snap-in

4.       The certificates for the Host computer can be in any of the following locations

                a.  Personal Certificates

                b.  Trusted People (if the host is W2K8)

                c.  Trusted Root Authorities (If the host is W2K3)

5.       In each store, expand the Friendly Name field and locate the certificate[s] for the Host server that have a Friendly Name starting with 'SCVMM_CERTIFICATE_KEY_CONTAINER' followed by either the FQDN / IP address / NetBIOS name of the Host server and delete them

6.       Re-add the host in SCVMM which recreates the certificates as needed.

 

 

MORE INFORMATION

SCVMM uses BITS to transfer payload between SCVMM managed computers. These data transfers are encrypted by using a self-signed certificate generated at the time a host machine is added to VMM. If these certificates are missing or corrupted from the VMM server or managed computers, the payload deployment job can fail.  Deleting the certificates and re-adding the host will cause the certificates to regenerate.

 

This informaiton has since been published under the following knowledge base article.

 

971264 Error 2912 when a Virtual Machine Manager 2008 operation fails due to a certificate issue: "Element not found (0x80070490)"
http://support.microsoft.com/default.aspx?scid=kb;EN-US;971264

One of the questions that we often receive is how to perform a P2V on an older Windows 2000 server.  The only way to accomplish this is to perform an Offline P2V where the source is restarted into the Windows Preinstallation Environment (WinPE) before VMM converts the physical disks to virtual hard disks (VHDs).  Most often this also means that the physical hardware running Windows 2000 server is also several years old.  This often introduces the challenge of finding a driver for the NIC for Vista/W2K8 since VMM 2008 uses the version of WinPE included on W2K8.  And since the server itself may be several years old finding the right driver can sometimes prove difficult. 

The following steps assume this same scenario.  I was trying to perform an Offline P2V of an older Windows 2000 server that was using a HP NIC3163 Fast Ethernet NIC. 

       1.     Collect the setupapi.app.log file from 'x:\windows\inf' while booted into WinPE. 

 

2.     Open the log file and navigate to the bottom and search up for the failure.  Note the very clear error message that "there are no compatibale drivers for this device"

 

            < excerpt from setupapi.app.log >

 

               >>>  [DIF_SELECTBESTCOMPATDRV -            

               PCI\VEN_8086&DEV_1229&SUBSYS_B1340E11&REV_08\3&13C0B0C5&0&28]

               >>>  Section start 2009/03/24 14:31:58.828

               cmd: wpeinit

               dvi: No class installer for 'Ethernet Controller'

               dvi: No CoInstallers found

               dvi: Default installer: Enter 14:31:58.828

               dvi:      {Select Best Driver}

               !    dvi:           Selecting driver failed(0xe0000228)

                dvi:      {Select Best Driver - exit(0xe0000228)}

               !    dvi: Default installer: failed!

               !    dvi: Error 0xe0000228: There are no compatible drivers for this device.

               <<<  Section end 2009/03/24 14:31:58.828

               <<<  [Exit status: FAILURE(0xe0000228)]

 

 

3.     Verify the PnPID of the installed NIC by collecting the NFO file from the server.  Use the following steps to collect this file and locate the PnPID of the installed NIC.

            a.         Click Start, point to Run and type "msinfo32" (without the quotes) in the Open box and click Ok.

            b.         This will open the System Information dialog.

            c.         Under System Summary, expand Components and then Network followed by Adapter.

            d.         In the right hand pane, a list of all the installed adapters is displayed.

            e.         Scroll down to locate the physical adapter and note the PnPID.

            f.          This PnPID should match the one listed in the error from the setupapi.app.log.

            g.         This should look similar to the following:

 

               Name [00000008] HP NC3163 Fast Ethernet NIC

               Product Name HP NC3163 Fast Ethernet NIC

               PNP Device ID   PCI\VEN_8086&DEV_1229&SUBSYS_B1340E11&REV_08\3&13C0B0C5&0&28

 

4.     Next, locate the correct driver for this NIC from the hardware vendor.  A quick search for the NIC used in the example turned up the following downloads for this NIC on HP's site.

 

http://h20000.www2.hp.com/bizsupport/TechSupport/DriverDownload.jsp?prodNameId=407753&lang=en&cc=us&prodTypeId=329290&prodSeriesId=407751&taskId=135

 

 

5.     Now we need to locate the correct driver from the hardware vendors site to provide for WinPE for the offline conversion.  Since VMM2008 uses the version of WinPE from Windows Server 2008, you will want to first locate W2K8/Vista drivers for the installed adapter.

 

6.     Here are the drivers available from the hardware vendor for this particular NIC.  Note the absence of W2K8/Vista drivers.

 

                        Microsoft MS-DOS

                        Microsoft Windows 2000

                        Microsoft Windows NT 4.0

                        Microsoft Windows Server 2003

                        Microsoft Windows Server 2003 for 64-bit extended systems

 

 

7.     Here is where we're going to run into a bit of trouble with this particular NIC.  The NIC is pretty old and the vendor doesn't provide any drivers for Vista or W2K8.  We're going to have to try some older drivers and hopefully one of those will work.  But it will be trial and error.  And it may end up that none of them are compatible with WinPE.

 

8.     I selected the latest released driver for this adapter which happens to be for Microsoft Windows Server 2003 (cp009602) and downloaded to the C:\HP directory.  The contents of the driver download may be extracted by double clicking on it.  There will be an extract option.  It's always a good idea to extract the contents to a different folder than where the actual executable is.

 

               Here are the extracted contents to C:\HP\Flat folder

                             cp009602.xml

                             cp_data.xml

                             cpqsetup.exe

                             install.xml

                             n100w32.sec

                             n100325.sys

                             netcpqi.ini

                             nicdrvd.dll

 

 

9.     Now we need to copy the contents of C:\HP\Flat to 'C:\Program Files\Microsoft System Center Virtual Machine Manager 2008\Driver Import' directory.  Create a logical folder structure under the 'Driver Import' folder such as ...NIC\HP NC3163.

 

10.  Then we're ready to start the P2V wizard again in SCVMM.

 

11.  Select the Offline conversion.  This is on the same screen where you select the drives to convert.  This option is located near the lower left hand corner on this screen.  Once expanded, select 'Offline conversion' and Ok.  The following screen is called 'Specify offline conversion options'.  The default option is 'obtain IP automatically'  but you'll want to change that to 'use the following IPv4'.  This will automatically pull the IP information from the source and display it in a window that can be edited.  Verify this information is correct and continue forward and complete the P2V wizard. 

 

If the driver selected was the correct one, then P2V process should complete without any further problems.  If it wasn't, then follow the same steps outlined above with another of the drivers from the hardware vendors site.  Hopefully one will work with this version of WinPE and allow the P2V conversion to complete as expected. 

 

Another option is to temporarily install another newer NIC in the server as a workaround just to get the server virtualized. 

 

 

This information has since been published under the following knowledge base article.

 

970921 How to add NIC drivers for offline P2V in VMM2008
http://support.microsoft.com/default.aspx?scid=kb;EN-US;970921

So, you've ran across an issue with SCVMM that you're going to need some help with.  Great, that's what we're here for in the Virtualization space here at Microsoft.  We're ready, available and anxious to partner with you to figure out the underlying root cause of the problem.  To expedite this process, we've created a site to help you gather the necessary information that the support professional will need to quickly analyze the problem.  You'll also notice the solutions to several common problems that our customers report.  Hopefully one of these will help you resolve the issue at hand, but if not, please collect follow the steps outlined on the site to collect the required data and you'll be well on your way to working through the issue.

Here is a link to the site:  http://blogs.technet.com/scvmmcallback/

If you have any questions about the site, or comments in general, please feel free to send me a quick email and I can get the site updated.

You might also want to check out the following user forums on SCVMM.

http://forums.microsoft.com/TechNet/default.aspx?ForumGroupID=489&SiteID=17

UpgradeAdvisorReleased April 14, 2009, a number of Hotfixes have been rolled up into a single download. I strongly recommend this update for all SCVMM Servers. This update may require a reboot. This update replaces ‘KB959596 - Description of the System Center Virtual Machine Manager 2008 update to address physical to virtual (P2V) issues.' You should still install updates from KB962941. Best of luck!

Description of the hotfix rollup package for System Center Virtual Machine Manager 2008: April 14th, 2009
http://support.microsoft.com/kb/961983

This update will be provided automatically by 'Microsoft Update,' or it can be manually downloaded here. If installing manually, open an elevated command prompt and execute (without the quotes): “msiexec /update vmmServer64Update.msp BOOTSTRAPPED=1”

Issues Addressed
  VMware virtual machine migration
  Hyper-V VMs showing “unsupported cluster configuration”
  Differencing disk issues
  Agent status on Cluster nodes
  Replication software causing “unsupported cluster configuration"
  Error 2951 with VMware Virtual Center 


Summary

Install updates from KB962941 and KB961983
KB959596 is no longer necessary if KB961983 is installed.

Recommended hotfixes for System Center Virtual Machine Manager 2008
http://support.microsoft.com/kb/962941


Notes

This post was authored by Jonathan Jordan and is available on his site.  But important enough to be repeated here.  Install this update :)  Ok? 

 

Occasionally I find it necessary to capture a Dbgview trace from a Core installation of Windows Server 2008.  This commonly occurs as more and more people use a Core server as their VMM Host in either a P2V or V2V scenario.  When troubleshooting such an issue, CSS may need to capture Dbgview traces from all three servers involved (VMM server, host and source).  What follows are the instructions to capture a Dbgview trace from a Windows Server Core installation.  For general instructions on how to setup/configure a Dbgview trace, please refer to Cheng’s blog.

Steps to capture Dbgview trace on a Core server

a.  Copy the folder with all the Dbgview files to a local folder off the root of C drive like C:\Dbgview

b.  From an administrator’s command prompt change to the C:\Dbgview directory

c.  Type odson.reg (if this fails, add the registry entry manually)

d.  Type odsflags.cmd 255 (or 0xffffff alternatively)

e. Restart services:

                net stop vmmservice

                net stop vmmagent

                iisreset

                net start vmmservice

                net start vmmagent

f.  Type dbgview.exe

g.  Dbgview should start capturing data

However, on some Windows CORE installations the following error message then appears.

clip_image001

The obvious answer is to add the following privileges to the logged on user:

  1. Load and unload device drivers
  2. Debug programs

But how do you add those rights?  Since we don’t have any UI in Core there are two ways that I know of that you can configure this.

  1. Create a new template with these rights and apply to the Core server.
    1. On a non-Core server, open a new mmc
    2. Load the Security Templates snap-in 
    3. Right click on Security Templates, and select New
    4. Name the new template Debug
    5. Expand Debug, then Local Policies followed by User Rights Assignments
    6. Modify the necessary rights
    7. Save the new template
    8. Copy the template to the Core server
    9. Use the Secedit /configure to apply template

I’ve included an example of a properly configured secedit command.

secedit /configure /db hisecws.sdb /cfg hisecws.inf /overwrite /log hisecws.log

In this example, the db hisecws.sdb is used.  However you can use really any database because we’re only using it for a very brief amount of time and not making any changes to it.  We merely need the place holder.  The hisecws.inf would be the name of the template created above (Debug.inf).

The following link talks more about using the SECEDIT command.

http://technet.microsoft.com/en-us/library/cc737638.aspx

2. Another option is to either modify the default domain policy or create a new group policy object that includes these rights and apply to the OU where the Core server resides.  Then either wait for policy to refresh or run gpupdate /force to immediately apply the new policy.  This may be the easier way to go but not sure if this is possible in your environment.

This is part of the fun with running Server Core J

3. Another alternative would be to take the Host out of the equation altogether and simply try the P2V conversion to the VMM server directly as a test. 

More Posts Next page »
 
Page view tracker