A colleague of mine has posted a great article on one cause of these errors. Turns out there are specific refreshers that can be targeted providing WinRM and SCVMM the extra time needed to complete the job. Be sure to read the entire article carefully as there are implications with upgrading to SCVMM 2008 R2. Great article Robert!

Error in SCVMM

Error (2927)
A Hardware Management error has occurred trying to contact server <your host name comes here>.

(Unknown error (0x80338029))

As it turned out, during refresh we did not wait long enough for a wsman response. The operation that timed out was to enumerate the SAN LUNs.

In SCVMM 2008 this timeout is 120 seconds by default. Fortunately, we can increase this timeout to a maximum of 300 seconds.

Read the entire article:

SCVMM 2008 Error (2927) A Hardware Management error has occurred trying to contact server
http://blogs.msdn.com/robertvi/archive/2009/10/23/scvmm-2008-error-2927-a-hardware-management-error-has-occurred-trying-to-contact-server.aspx

...

Hyper-V Issue that can affect SCVMM operations

What: A new article has been released with a hotfix to resolve an issue where Windows 2008 R2 servers running on AMD Family 10h processors may restart unexpectedly or stop responding. No error messages are generated.

When: Issue may occur when the Live Migration feature is used to move a virtual machine (VM) from one Windows Server 2008 R2 Hyper-V based node to another. This implies that the nodes are members of a Failover Cluster. This problem usually affects the source node that is the original location of the VM.

Why: Erratum 383 in AMD Family processors generates a machine check exception.

Hotfix Information: To apply the hotfix the computer must be running Windows Server 2008 R2. You may have to restart the computer after installing the hotfix.

SCVMM Note: Install this hotfix on all Windows Server 2008 R2 systems that are nodes of a Failover Cluster.

Microsoft Support URL

The computer stops responding or restarts during the Hyper-V Live Migration process in Windows Server 2008 R2
http://support.microsoft.com/default.aspx?scid=kb;EN-US;981618

…

A new KB article regarding Red Hat is being posted on Microsoft Support any day now. For those of you who do not need the official KB, here’s the content. This is a relatively minor fix. If you are not a SQL expert, find one, or just follow the simple instructions for the free SQL Studio Express Edition. Thanks!

URL when published http://support.microsoft.com/kb/2022557.

Symptoms

On System Center Virtual Machine Manager 2008 R2, it is possible to select Red Hat as the virtual machine 'Operating System' property for a virtual machine hosted by Hyper-V. The job, however, will show as failed with Error 10637. This can occur when creating a new virtual machine or changing this property on an existing VM. This does not affect the function or stability of the virtual machine itself. This property is found on the General tab of the virtual machine properties dialog.

Error (10637)

The virtualization software on host server.contoso.com does not support the Red Hat Enterprise Linux 5 operating system. 

Recommended Action

Specify a host with different virtualization software and then try the operation again.

Cause

A database entry that identifies the list of supported virtual machine operating systems per hypervisor requires an update. The following actions will remain unavailable for Red Hat virtual machines in SCVMM following this update:

• Shut down of the guest operating system from the SCVMM Administrator Console
• Detection and display of the VM Additions version for a virtual machine in the SCVMM Administrator Console and with Windows PowerShell
• Display of the computer name of the guest operating system

Resolution

Using SQL Server Management Studio Express, or a full SQL installation, make a full backup of the SCVMM database. Next, connect to the SCVMM database (VirtualManagerDB by default). Open a new query and enter the commands below. The number of affected columns will be returned in the 'Messages' window if present.

update tbl_IL_OS

set OSFlags=0x14

where Name like 'Red Hat Enterprise Linux 5%'

…

Here’s a tip I’d like to pass on that resulted solely from trial and error.

I regularly see Failover Cluster, Hyper-V, NIC Teaming, VLANs, iSCSI networks and so on as components of a complex environment managed by SCVMM. Each of these components can affect decisions regarding network design. In the process of implementing the desired settings there will often remain a network, or NIC, that goes unused – even if temporarily. This unused NIC may have had a cable plugged in at some time, but for now there is no cable and the NIC shows a status in Windows of ‘Network cable unplugged.’ This seems harmless enough, but I’ve found it to be a real problem generator. Luckily, this has a simple fix: right click and select ‘Disable.’

So, a NIC left enabled but without a cable plugged in can cause problems. Make this correction, then retry whatever issue you previously experienced. I have resolved cases involving WMI, BITS, WinRM and so on. You might see these jobs affected:

• Adding a Host
• P2V
• Live or Quick Migration
• Quick Storage Migration
• and so on…

SCVMM is not in any way more affected by an unplugged network cable than are other Microsoft products. SCVMM just touches many of these products and therefore can be affected in the same way they are. I hope this helps a few of you out there!

…

Another great find by Michael Michael. He has this written up beautifully, so I’ll just provide a link. You may be affected and need a resolution if you have these conditions:

• PRO Integration enabled
• Long load time for Admin Console
• VMMAdmin.exe has a large memory footprint

Reason and Solution provided here. Thanks again M2.

VMM Administrator Console taking too much time to load http://blogs.technet.com/m2/archive/2010/03/12/vmm-administrator-console-taking-too-much-time-to-load.aspx

…

In the last few weeks we have seen a few cases with an unusual error code: 0x800704DD. If you run into this issue the fix is simple. Our fearless leader Michael has posted a blog explaining the problem and how to fix it. Definitely worth a visit.

Direct Link: http://blogs.technet.com/m2/archive/2010/03/03/error-800704dd-happening-during-a-lan-migration-or-deployment-of-a-vm.aspx

…

A friend and co-worker discovered an interesting fix for a failed SCVMM P2V attempt involving Windows 2000 SP4. We’ve seen this same type of issue when converting Windows 2000 systems for years, and WMI is often part of the issue. Mike has provided an article listing three WMI updates that overcome many of the challenges we have seen. In the first day of this article’s publishing we fixed two cases involving P2V of Windows 2000. I can’t suggest strongly enough that you read his post and apply the hotfixes. Thanks Mike! Great contribution!

Jump to the article here.

The specific hex we have seen this most applicable to is 0x80041001, but not limited to only this error.

…

A great colleague has written a simple, well illustrated article on the purpose of CPU types for virtual machine’s in SCVMM. This will serve as my reference from now on. Link below. Share with your peers. This issue pops up all the time with no great explanation until now. Thanks Mark!

What is CPU Type in SCVMM 2008 R2 VM Processor Hardware Profile?

…

Description

The VMMCA is a diagnostic tool you can use to evaluate important configuration settings for computers that either are serving or might serve VMM roles or other VMM functions. The VMMCA scans the hardware and software configurations of the computers you specify, evaluates them against a set of predefined rules, and then provides you with error messages and warnings for any configurations that are not optimal for the VMM role or other VMM function that you have specified for the computer.

Requirements

You must install and run the VMMCA on the computer that either currently is or will become your VMM 2008 server. To review the system requirements for the VMM 2008 server, go to http://go.microsoft.com/fwlink/?LinkId=133516.

Prerequisite: Microsoft Baseline Configuration Analyzer (MBCA)

Note: The system requirements for the MBCA indicate that only Windows Server 2003 is supported. However, the VMMCA and the MBCA have been tested and are supported on Windows Server 2008 64-bit.

Where to get it

1) Download pre-req: Microsoft Baseline Configuration Analyzer (MBCA)
http://go.microsoft.com/fwlink/?LinkID=97952

2) Download VMMCA
http://www.microsoft.com/downloads/details.aspx?FamilyID=02d83950-c03d-454e-803b-96d1c1d5be24&displaylang=en

Common Usage Scenarios

The list of scenarios are not comprehensive, but will provide you with some common usage scenarios to provide you with an understanding of when to use VMMCA 2008.

Scenario 1 – I have a Windows Host that is in a not responding state.  Use VMMCA 2008 to check the host for commonly known issues.

Scenario 2 – A Windows host shows a status of “Needs Attention”.  Use VMMCA 2008 to check the this host for commonly known issues.

Scenario 3 – I am using SCVMM 2008 to manage a VMware environment, but I am unable to create new virtual machines on the VMware environment. Use VMMCA 2008  to evaluate the Virtual Center environment for commonly known issues.

Scenario 4 – I am using the integration between SCVMM 2008 and Operations Manager, but I am unable to see a particular VM or Host in the integrated mapping view.  Use VMMCA 2008 to check the host or virtual machine for an operations manager agent.

Scenario 5 – I am using SCVMM 2008 to do P2V conversions and the job is failing.  Use VMMCA 2008 to check the P2V source for commonly known issues.

Scenario 6 -  I tried to create a Virtual Machine and it failed during customization and installation of virtual guest services.  Use VMMCA 2008 on the Windows Host to find commonly known issues.

…

Overview

The SCVMM 2008 and 2008 R2 Microsoft Product Support Reports helps Support Professionals, IT Pros and System Administrators gathering critical system and logging information used in troubleshooting support issues. Built upon the success of earlier versions of SCVMM MPS Reports, this new release adds compatibility for SCVMM 2008 R2 and Windows 2008 R2 OS platform.

Where to get it

SCVMM 2008 and 2008 R2 MPS Reports are available at the  Download Center URL link here  http://www.microsoft.com/downloads/details.aspx?FamilyID=4e3dc013-c299-4b80-9735-64a8b9fa60c7

Data Collected

The Virtual Machine Manager (VMM) version of the MPS Reporting Tool gathers the following diagnostic data for the VMM server, the virtual machine hosts and library servers that the VMM server manages, and physical to virtual conversion (P2V) source computers:

• Virtual Machine Manager traces
• Virtual Machine Manager database
• Windows event logs and virtual server event logs
• IP address
• Virtual Machine Manager setup logs
• SQL Server setup logs
• Registry information
• Open ports information
• System Information (msinfo)

Basic Instructions

INSTALLING THE MPS REPORTING TOOL
=================================
>>>To install the MPS Reporting Tool

1. Download the file MPSRPT_SCVMM.exe.
2. Copy the file to the computer on which you want to diagnose problems.
3. Open an elevated Command Prompt window with administrative privileges, and then run MPSRPT_SCVMM.exe., or right-click MPSRPT_SCVMM.exe, and then click "Run as Administrator".
4. Accept the End-User License Agreement (EULA).

By default, the MPS Reporting Tool is installed in the %SystemRoot%\MPSReports\VirtualMachineManager folder.

…

I’ve seen a number of customers run into networking and general communication issues following a move to Windows 2008 R2. And to manage Windows 2008 R2 you must be running SCVMM 2008 R2. Both require checking for updates.

Windows 2008 R2 – Make sure the network card drivers are up to date… all of them. If the server vendor does not have drivers updated for Windows 2008 (or R2), check the site of the actual NIC manufacturer as most built-in NICs are simply rebranded. This will solve a majority of your networking issues off the bat. Make sure you update the drivers on the VMM Server as well as the Hosts.

SCVMM 2008 R2 – Did you upgrade from SCVMM 2008 to SCVMM 2008 R2? If so, you should check each Host in the SCVMM Admin Console to determine if the agents have been updated. If not, update them now by right clicking on them and selecting ‘Update Agent.’

If you have performed the actions detailed above and are still experiencing communication issues with Hosts you can uninstall the ‘Microsoft System Center Virtual Machine Manager Agent’ manually on the Hosts from ‘Control Panel> Programs and Features.’ After this, go back to the SCVMM Admin Console, remove the Host by right clicking on it and selecting ‘Remove Host.’ Once removed, add it back with the Admin Console by selecting ‘Add Host’ under Actions on the right side of the console. This will clear up any agent update issues.

This covers two of the most common issues seen lately. If still experiencing issues check some of the other posts on this site that address other possible reasons. Good luck, and enjoy the new product!

Free Evaluation Download!

…

Downloadable documents are now available for VMM R2!  Check them out on TechNet here:  http://go.microsoft.com/fwlink/?LinkId=162764

Available documents for download

• Deployment Guide
• Guide to Operations Manager Integration
• Security Guide
• Operations Guide
• Scripting Guide
• Cmdlet Reference
• Building PRO-Enabled Management Packs

Use the links in the following table to find documentation that you can download and print to help you deploy, operate, and maintain System Center Virtual Machine Manager VMM 2008 R2. You can also browse the product documentation online in the Technical Library.

Feedback on these? Please send it to: scvmfdbk@microsoft.com

…

Announcement straight from the source! Enjoy!

System Center Virtual Machine Manager 2008 R2 has RTM’d and GA via volume licensing is set for October 1. This is great news for all and I’d like to especially thank our VMM 2008 R2 Development, Product Management, and Test teams. Lots of hard work fueled by their passion in virtualization and management has resulted in a very good software release.

A 180-day evaluation version is now available, too, on the Microsoft Download site. You can access it here.

Please experience for yourself what the 10,000+ people who have previously downloaded our ‘Release Candidate’ plus organizations such as Continental Airlines, Lionbridge Technologies, and Indiana University have seen with VMM 2008 R2!

I encourage everyone to explore the new System Center Virtual Machine Manager 2008 R2 and its new features such as quick storage migration, live migration, and many others. We even offer support for vSphere 4.

To learn more on the new features and capabilities of VMM2008 R2, please try to attend our upcoming TechNet session ‘Technical Overview of System Center Virtual Machine Manager 2008 R2’. Presented by our Technical Product Manager Kenon Owens, it will be chocked full of new and cool VMM 2008 R2 items. Go here to register for this Wednesday, September 09, 2009 (10:00 AM Pacific) event.

Thanks!

Zane Adam

Senior Director, Virtualization and management

There are a number of reasons a Virtual Machine (VM) can fall into a status of ‘Unsupported Cluster Configuration.’ A brief list, pulled from TechNet, shows that storage, networking, and other factors can cause this status. (Source: http://technet.microsoft.com/en-us/library/cc967323.aspx)

• The virtual machine is on a LUN that contains more than one virtual machine.
• The virtual machine is using non-clustered storage.
• One or more virtual network adapters on the virtual machine are not connected to a highly available virtual network.
• An ISO image is attached to highly available virtual machines.
• A VMware HAVM is connected to a port group that does not exist on all nodes of the host cluster.

This article deals with the third bullet, which effectively means, ‘settings are not identical on all virtual network (VN) adapters.’ Here are the TechNet instructions. Next we’ll discuss checking these automatically.

For a virtual network to be considered common by VMM and available to highly available virtual machines on a host cluster, each virtual network in the host cluster must meet the following requirements:

• The virtual network name must be identical on each host in the cluster. Virtual network names are case-sensitive, so the cases of all characters must match.
• The host network adapters to which the virtual network is attached on each host in the cluster must have the same location.
• The virtual network must have the same tag on each host in the cluster.

After you update the virtual network configurations on all nodes, refresh the cluster to ensure that each virtual network is detected as common. Then check the Networks tab in the host cluster properties to verify that the networks have been added to it.

So, check the Network Name, Location and Tag for inconsistencies. Once you have done this for all VNs on each Host in the Host Cluster, corrected any inconsistencies and refreshed the Host Cluster, the VMs will change back to a healthy status. Verifying all of this can take some time, and is prone to human error as these settings are case sensitive and take into account spaces. Instead, why not run a PowerShell script that will report all settings for you? (Scroll to the bottom to download the script).

Here are the three settings that must be consistent on all nodes.

Here’s the PowerShell script we’re going to use.

The script is saved as ‘VMNicInfo.ps1’ in the VMM Library. This way, it can be run from within the Admin console. I’ve created a folder named ‘Scripts’ to place it in. Refresh the library to see it.

Now we run the script by right clicking on it and answering a few prompts. Enter ‘r’ to run the script if prompted. Next, enter the name of your Host Cluster and press Enter. (This script must be run from an Admin console running on the VMM Server. To run it from the Admin console on a different system change ‘localhost’  to the name of your VMM Server and surround it with quotes.)

You see in the image above that the Name and Location are the same. There is no entry for Tag on either VN, so they match as well. In the example below we add a value to Tag on one of the systems.

Now we run the script again and see that the output reflects this change. (If this does not work for you make sure you have refreshed the Host Cluster, in the red box). You will also notice that the VMs running on the Host Cluster change to a state of ‘Unsupported Cluster Configuration.’

Change the Tag back to its previous value (nothing in my case) and perform another refresh to see your VMs change back to a healthy state.

That’s it. I hope this clears up any questions or misconceptions you may have had. Below is the script. Be sure to copy this to Notepad as it will remove all formatting characters. Or, download the script here, then run from a VMM PowerShell prompt.

#####################################################################

function DisplayNicInfo($VMHostName)

{

  $yy= get-VirtualNetwork -VMHost $VMHostName;

  $yy | ForEach-object {write-host "    Name      " $_.Name;

                        write-host "    Locations " $_.Locations;

                        write-host "    Tag       " $_.Tag;

                       }

}

#####################################################################

$clusname = read-host "Host Cluster name to check"

Write-Host ""

$VMMServer = get-vmmserver -computername localhost

$Cluster = get-vmhostcluster -name $clusname

$VMHosts = get-vmhost -vmhostcluster $Cluster

$VMHosts | ForEach-object {Write-Host "VMHost: " $_.Name;

  DisplayNicInfo($_.Name);

  Write-Host ""}

Big thanks to Austin, who reworked the script into a thing of beauty. Go Perf! (http://blogs.technet.com/askperf)