Adding a Host to System Center Virtual Machine Manager 2008 (SCVMM 2008) fails with a variation of Error (2916):
Error (2916) VMM is unable to complete the request. The connection to the agent ServerA.contoso.com was lost. (The WinRM client cannot complete the operation within the time specified. Check if the machine name is valid and is reachable over the network and firewall exception for Windows Remote Management service is enabled. (0x80338126)) Recommended Action: Ensure that the WS-Management service and the agent are installed and running and that a firewall is not blocking HTTP traffic. If the error persists; reboot ServerA.contoso.com and then try the operation again.
Error (2916) VMM is unable to complete the request. The connection to the agent ServerA.contoso.com was lost. (The WinRM client cannot complete the operation within the time specified. Check if the machine name is valid and is reachable over the network and firewall exception for Windows Remote Management service is enabled. (0x80338126))
Recommended Action: Ensure that the WS-Management service and the agent are installed and running and that a firewall is not blocking HTTP traffic. If the error persists; reboot ServerA.contoso.com and then try the operation again.
Specific content is being filtered by a non-Windows firewall. The firewall could be software installed on either the SCVMM 2008 Server or the Host that is being added. More likely, there is a hardware appliance firewall on the network between the two communicating servers.
Test multiple communication protocols between the two systems; the SCVMM 2008 Server and Host in this example. Some firewalls can have content filtering enabled despite showing that it is not. Remove all non-Windows software firewalls and bypass all hardware appliance firewalls entirely long enough to perform testing to verify whether or not they are contributing to the problem.
The following tests are examples of protocols that should always succeed. Test both directions always:
WinRM successful reply example:
C:\>winrm id -r:ServerA IdentifyResponse ProtocolVersion = http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd ProductVendor = Microsoft Corporation ProductVersion = OS: 6.1.7600 SP: 0.0 Stack: 2.0
Recently a firewall appliance sold by a major vendor showed content filtering disabled and not licensed to be turned on, yet was still filtering specific content. This was discovered through examination of network traces. Do not assume content, protocols or traffic are not being blocked. Perform tests to verify.
=====
For the latest version of this article see the link below:
KB2465160 - Add Host or other action fails with (2916) 0x80338126 in System Center Virtual Machine Manager 2008
J.C. Hornbeck | System Center Knowledge Engineer
The App-V Team blog: http://blogs.technet.com/appv/ The WSUS Support Team blog: http://blogs.technet.com/sus/ The SCMDM Support Team blog: http://blogs.technet.com/mdm/ The ConfigMgr Support Team blog: http://blogs.technet.com/configurationmgr/ The SCOM 2007 Support Team blog: http://blogs.technet.com/operationsmgr/ The SCVMM Team blog: http://blogs.technet.com/scvmm/ The MED-V Team blog: http://blogs.technet.com/medv/ The DPM Team blog: http://blogs.technet.com/dpm/ The OOB Support Team blog: http://blogs.technet.com/oob/ The Opalis Team blog: http://blogs.technet.com/opalis
Just a heads up on a new KB article we published on how to determine if a cluster is over-committed in System Center Virtual Machine Manager 2008. If you do any kind of admin work with System Center Virtual Machine Manager 2008 you'll want to add this to your favorites.
The cluster status displayed in the System Center Virtual Machine Manager 2008 Administrator's Console may change from "OK" to "over-committed" after a cluster refresh operation completes.
Note : The host status values do not change in the VMM Administrator Console until the VMM server performs a host refresh, which runs automatically every 30 minutes. You can run a refresh on demand by right-clicking the host and then clicking Refresh.
When the displayed status of a managed cluster becomes over-committed then the administrator is not able to use this cluster for placement of virtual machines for new or migrated virtual machines.
This occurs because the sum of free slots in the entire cluster is greater than the sum of slots in the largest host (both free and used).
The resolution will depend on a variety of factors. The primary factors are
· The cluster reserve value defined in SCVMM console · The amount of memory in each cluster node · The amount of memory assigned to each VM · The placement of the VMs on nodes in the cluster
Below you will see a discussion on how to determine if a cluster is over-committed. This will help to demonstrate how the factors mentioned above play a role in a cluster showing as over-committed.
In some cases, the resolution might be as simple as ensuring an even distribution of VMs on each cluster node or ensuring all nodes have same amount of RAM. A more advanced resolution might require considering memory requirements of VM’s and placing VMs with similar memory requirements in the same cluster.
How VMM determines if a cluster is over-committed:
1. Find out the HAVM with the largest allocated memory across all nodes in the cluster. The allocated memory of this VM represents the size of a slot.
2. Calculate number of “used slots” on each host:
Note : More than one virtual machine can be used to fill a slot. For instance, if the slot size is 8 GB, two virtual machines with 4 GB RAM each can be used to fill one slot.
a. Using the slot size determined in step 1, group as many VMs as possible into a slot based on their allocated memory.
i. For example: Consider case of three VM’s with 2,4 and 8 GB of memory allocated and a slot size of 8GB. 2 slots would be required to group the three VM’s without exceeding the slot size.
b. Continue the process until all VM’s have been grouped into a slot.
c. The number of groupings will be the number of “used slots"
3. Calculate number of “free slots” on each host:
a. Determine the physical memory on the host
b. Determine the host memory reserve defined in SCVMM for each node
c. Determine the memory allocated to each VM
d. Plug the values into the following formula and divide by the slot size determined in step 1.
(PhysicalMemory – HostMemoryReserve – VMMemory) / SlotSize
4. Determine the number of slots that need to be in reserve.
The cluster reserve, R, defines the number of nodes that we must protect against failing. By summing the number of “used slots” and “free slots” on the R largest host[s] we are able to the determine number of slots to be held in reserve.
5. Determine if the Cluster is over-committed. As long as the number of “free slots” in the entire cluster (summation of # obtained in step 3) is greater than the slots that need to be in reserve (step 4), the cluster is not overcommitted.
Example: Overcommitted formula implementation Cluster name: VMM-Cluster1 Cluster reserve = 1 Cluster nodes: All cluster nodes run on identical hardware with 32 GB RAM each N1: VMM-ClusterN1 N2: VMM-ClusterN2 N3: VMM-ClusterN3 N4: VMM-ClusterN4 Virtual Machines on this cluster: 16
Note : This example assumes that the default value for the cluster reserve of 512kb. It is common for this value to be increased and subsequently impacts this calculation.
6. Find the HAVM with the largest allocated memory to define the slot size. Using the above table as our example, we see the largest memory allocated for any VM is 8GB. This represents the slot size for this cluster at this time.
7. Calculate number of used slots on each host:
Based off of an 8GB slot size and being able to fit more than one VM per slot until reaching the 8GB maximum, we determine the number of “used slots” per host (see above table for details):
VMM-ClusterN1: 3 VMM-ClusterN2: 3 VMM-ClusterN3: 2 VMM-ClusterN4: 1
8. Calculate number of “free slots” on each host: (PhysicalMemory – HostMemoryReserve – VMMemory) / SlotSize
Note : We can’t have a partial slot. If the formula returns 1.8 slots, then use a value of 1.
VMM-ClusterN1: 1 slots (32GB - 512kb - 20GB in use / 8GB) VMM-ClusterN2: 1 slots (32GB - 512kb - 18GB in use / 8GB) VMM-ClusterN3: 2 slots (32GB - 512kb - 10GB in use / 8GB) VMM-ClusterN4: 2 slots (32GB - 512kb - 8GB in use / 8GB)
9. Determine the number of slots that need to be in reserve
Cluster Reserve is 1, so we look for 1 largest host(s). In our example we determine nodes 1, 2 and 3 represent the largest total slots per node. We will use VMM-ClusterN1 as the host that represents the total number of slots that must be held in reserve
10. Determine if the Cluster is over-committed
Add up the free slots of remaining hosts:
VMM-ClusterN2: 1 VMM-ClusterN3: 2 VMM-ClusterN4: 2 Total: 5
As long as the number of free slots in the entire cluster (summation of number obtained in step 3 minus the largest host) 5 is greater than sum of slots in the largest hosts (both free and used) 4, the cluster is not overcommitted.
HAVM Placement and “Over-committed” Status
Cluster reserve is a unique feature of VMM 2008 and VMM 2008 R2.The cluster reserve specifies the number of node failures a cluster must be able to sustain while still supporting all virtual machines that are currently deployed on the clustered hosts. If a host cluster cannot withstand the specified number of node failures and still keep all of the virtual machines running, the cluster is placed in an Over-committed state.
For example, if you specify a cluster reserve of 2 for an 8-node cluster, the rule is applied in the following ways:
When placing a virtual machine in a failover cluster, the placement process in VMM calculates whether the new virtual machine will over-commit the cluster. If the action will over-commit the cluster, the cluster hosts are not made available for placement.
Note : An administrator can override this and place a VM on a host in an over-committed cluster during manual placement.
VMM’s cluster refresher updates the host cluster’s over-committed status after each of the following events:
The cluster reserve is set on the General tab of the host cluster properties. For a procedure, see How to View and Modify the Properties of a Host Cluster (http://go.microsoft.com/fwlink/?LinkID=162986).
KB2463008 - How to determine if a cluster is over-committed in System Center Virtual Machine Manager 2008
Our very own Jonathan Jordan just posted a great article describing a System Center Virtual Machine Manager 2008 support incident he worked recently where a small percentage of machines were failing during the Physical to Virtual (P2V) conversion process. He walks through the steps he took to identify the cause and resolve the issue and shows you how you can do the same in your environment. Whether you've had to troubleshoot the SCVMM 2008 P2V process or not, you won't want to miss this one.
Physical to Virtual (P2V) conversion is one of the most popular features of SCVMM. This said, there are a plethora of articles written to make the process easy to understand, or to resolve common problems. Last week I worked with an company that had converted around three hundred systems (nice!). Two systems were giving them a headache, so they reached out to Microsoft Support. Since they had successfully converted so many systems already, I knew the problem was almost certainly NOT with the SCVMM server or any Destination Hosts (managed Hyper-V systems) they had been using. What does this leave? The two P2V Source machine (ones to be converted). In the end this proved true…
To continue reading Jonathan's article click here.
When using System Center Virtual Machine Manager 2008 (SCVMM 2008), a Physical to Virtual (P2V) conversion of a Windows Server 2003 system fails due to the Volume Shadow Copy Service (VSS) on the P2V source machine. You will receive an error similar to the following:
Error (3110) VMM is unable to create a snapshot set on win2k3.contoso.com. An unexpected error occurred during the Volume Shadow Copy service operation. (Unknown error (0x80042318))
Recommended Action Try the operation again.
Additionally, attempts to list the VSS writers on the P2V source computer produces no output. This test produces an error in the application event log with errors similar to the one below:
Event Type: Error Event Source: VSS Event ID: 12302 Description: Volume Shadow Copy Service error: An internal inconsistency was detected in trying to contact shadow copy service writers. Please check to see that the Event Service and Volume Shadow Copy Service are operating properly.
Note: To list the VSS writers from a command prompt run the following command:
vssadmin list writers
This can occur because a corrupted subscription is detected by the COM+ system causing it to stop processing further VSS providers. COM+ is essential to the Volume Shadow Copy Service.
A Hotfix is available that corrects this issue. This hotfix is already included in Service Pack 3 (SP3) for Windows Server 2003 and requires that Windows 2003 SP1 or SP2 is already installed. The hotfix is also available separately at the link below:
KB968447: The COM+ Event System stops processing the query for matching subscriptions when it detects a corrupted subscription on a Windows Server 2003-based computer (http://support.microsoft.com/default.aspx?scid=kb;EN-US;968447)
Similar issues may also be resolved by following the article below.
KB940184: Error message when you run the "vssadmin list writers" command on a Windows Server 2003-based computer: "Error: 0x8000FFFF" (http://support.microsoft.com/default.aspx?scid=kb;EN-US;940184).
KB2459566 - P2V conversion fails with error 3110 in System Center Virtual Machine Manager 2008
The Microsoft System Center team is pleased to announce the launch of System Center Virtual Machine Manager Self-Service Portal 2.0 (SSP).
Download the SSP Now!
The System Center Virtual Machine Manager Self-Service Portal 2.0 (SSP) is a fully supported, partner extensible solution that enables customers to dynamically pool, allocate, and manage their compute, network and storage resources to deliver a private cloud platform in their datacenter. Learn more.
Using System Center Virtual Machine Manager Self-Service Portal 2.0 (SSP), you can respond more effectively—and at a lower cost—to the rapidly changing needs of your organization. Build on top of Windows Server 2008 R2, Hyper-V and System Center VMM, it enables you to offer infrastructure as a service.
It includes a pre-built, role-based self-service portal for both the datacenter managers and business unit IT consumers – all in a dynamic running engine. The portal reduces time needed to provision infrastructures and their components by offering a business unit “on-boarding” infrastructure request and change management. It also includes detailed guidance on how to implement the portal inside your environment. SSP is freely available, and fully-supported by Microsoft.
Key Features:
· Automation and Guidance: To assess, plan and design your private cloud foundation infrastructure
· Customer/business unit on-boarding: Automated workflows to onboard business unit IT departments onto to your virtualized shared resource pool
· Dynamic provisioning engine: To rapidly provision virtualized infrastructure in conjunction with System Center and Hyper-V
· Self-Service portal: To empower consumers of IT to request and provision infrastructure for their apps/services
· Partner Extensibility: Enable partners to expose their unique hardware capabilities through familiar Microsoft scripting technologies while providing variety and flexibility to IT
Enjoy!
Mike Briggs | Senior Support Escalation Engineer