Jonathan's Virtual Blog : BITS

Updates! Updates! SCVMM, Hyper-V, Cluster

Jonathan — Mon, 29 Dec 2008 22:23:00 GMT

There are many updates for SCVMM and Hyper-V. Some are required just following product install, others are provided to offer greater functionality. Some updates/hotfixes address issues that may constitute a headache for you down the road. I'll list everything I know and provide recommendations. This post will change over time as new updates and issues arise. If you find a hotfix or update that you think should be posted here, please let me know. Include the environment conditions (Windows version, etc) and what the update addresses. Here's to being proactive!

Article Ratings

A – Required by product to operate correctly

B – Resolves known issues

C – Adds Functionality

JJ – Get it. This is on my short list when working with customers.

A – SCVMM – Hyper-V

956589 Description of the Hyper-V update for issues that may occur when you manage the Hyper-V role on the 64-bit editions of Windows Server 2008 by using SCVMM

http://support.microsoft.com/default.aspx?scid=kb;EN-US;956589

A – SCVMM | BITS

956774 A Background Intelligent Transfer Service (BITS) client cannot handle files that have paths that contain the volume GUID in Windows Server 2008 or in Windows Vista

http://support.microsoft.com/default.aspx?scid=kb;EN-US;956774

B – JJ – SCVMM | WMI

958124 A wmiprvse.exe process may leak memory when a WMI notification query is used heavily on a Windows Server 2008-based or Windows Vista-based computer

http://support.microsoft.com/default.aspx?scid=kb;EN-US;958124

B – JJ – SCVMM | WMI

954563 Memory corruption may occur with the Windows Management Instrumentation (WMI) service on a computer that is running Windows Server 2008 or Windows Vista Service Pack 1

http://support.microsoft.com/default.aspx?scid=kb;EN-US;954563

B – JJ – SCVMM - WMI

955805 Certain applications become very slow on a Windows Server 2008-based or Windows Vista SP1-based computer when a certificate with SIA extension is installed

http://support.microsoft.com/default.aspx?scid=kb;EN-US;955805

B – JJ – SCVMM | P2V

959596 Description of the System Center Virtual Machine Manager 2008 update to address physical to virtual (P2V) issues

http://support.microsoft.com/default.aspx?scid=kb;EN-US;959596

B – JJ – Hyper-V | VSS

956697 Windows Server 2008 Hyper-V VSS writer is not used during a backup job because of corrupted or invalid virtual machine configuration files

http://support.microsoft.com/default.aspx?scid=kb;EN-US;956697

B – Hyper-V | VSS

959978 Error message when you back up a Windows Server 2003-based virtual machine on a Windows Server 2008 Hyper-V-based computer: "GetWriterStatus FAILED for Selected writer [Microsoft Hyper-V VSS Writer], writer is in state [9] [VSS_WS_FAILED_AT_FREEZE]"

http://support.microsoft.com/default.aspx?scid=kb;EN-US;959978

B – Hyper-V |SCVMM

957967 Stop error message on a Windows Server 2008-based computer that has the Hyper-V role installed: "STOP 0x0000001A"

http://support.microsoft.com/default.aspx?scid=kb;EN-US;957967

B – Cluster | Hyper-V

957311 Recommended hotfixes for Windows Server 2008-based server clusters

http://support.microsoft.com/default.aspx?scid=kb;EN-US;957311

B – JJ - Hyper-V | Drive Letters

958184 Virtual machine backup operations fail in Windows Server 2008 when Hyper-V virtual machine files are saved on a volume that is mounted on a failover cluster by using a volume GUID

http://support.microsoft.com/default.aspx?scid=kb;EN-US;958184

B – Hyper-V | Cluster

958065 You cannot configure a Hyper-V virtual machine by using Windows Server 2008 Failover Clustering when the virtual machine uses a storage device that is managed by a third-party clustered file system or a third-party replication solution

http://support.microsoft.com/default.aspx?scid=kb;EN-US;958065

C – Hyper-V | Cluster

951308 Increased functionality and virtual machine control in the Windows Server 2008 Failover Cluster Management console for the Hyper-V role

http://support.microsoft.com/default.aspx?scid=kb;EN-US;951308

Technorati Tags: microsoft,scvmm,hyper-v,virtual,virtualization,windows

How to Troubleshoot Slow BITS Performance, Hosts 'Not Responding' and 'Needs Attention' Communication Issues

Jonathan — Mon, 29 Dec 2008 15:37:00 GMT

You find yourself with one of the following situations

1. Deployment or migration of data utilizing LAN (not SAN Transfer) is very slow in general

2. BITS is slow in one direction only: Fast from Host01 to Host02, but slow from Host02 to Host01 for example

3. Host reports in SCVMM Admin Console "Host not responding" or "Needs attention"

4.     Under 'Status' of the Host the following may show "Not Responding"
    - Connection status
    - Agent Status

All of these issues can be related to networking components. Three common tasks can be performed to address this

1. Test for BITS errors

2. Verify GPO settings

3. Correct vendor specific and Windows system settings for ‘Task Offloading’ (This item will likely have the greatest effect)

Additional Items that just make sense

Verify all network settings on all involved SCVMM and Hyper-V systems. Set the network speed on the NICs and switches. Do not leave as auto negotiate. Also, remove all NIC teaming software. Network Card drivers currently do not always play nice with virtualization technologies. (Uninstall the teaming software. Disabling will not prevent drivers from loading at startup). If you cannot remove your teaming software at this time, at least make sure the namespace is available by typing ‘wmic nic’ at an elevated command prompt. If you get a screen full of data, you may be ok. If you get any kind of error, yank teaming software now and reboot.

1 - BITS Errors
Determine if BITS (Background Intelligent Transfer Service) is encountering errors. Attempt a P2V/Migration/Deployment transfer with SCVMM. While the transfer is occurring between servers, run the following commands on both servers:
Bitsadmin /list /allusers /verbose >bitsadmin.txt

Example BITS output below shows how long the transfer lasted (MODIFICATION TIME minus CREATED TIME). Retry delay shows that there were no errors. This is a clean run. If there are errors review the system and application event logs for errors that are BITS related. Involve your network team as well for a trace.

Example output…

BITSADMIN version 1.0 [ 5.1.2600.0 ]
BITS administration utility.
(C) Copyright 2000-2001 Microsoft Corp.

GUID: {BA568D81-B234-47F2-9BA3-686743F89794} DISPLAY:
https://cotoso.com:443/8272f226d8b8cd818406b28de9b6ad0f012b9a55/C:A5cProgramDataA5cM
icrosoftA5cWindowsA5cHyperVA5calbtest1A5cvmguest.iso/E:A5cVirtualMachinesA5calbtest
1A5cvmguest.iso
TYPE: UNKNOWN STATE: TRANSFERRED OWNER: NT AUTHORITY\SYSTEM
PRIORITY: FOREGROUND FILES: 1 / 1 BYTES: 42948608 / 42948608
CREATION TIME: 12/3/2008 9:04:48 AM MODIFICATION TIME: 12/3/2008 9:16:44 AM
COMPLETION TIME: 12/3/2008 9:16:44 AM
NOTIFY INTERFACE: UNREGISTERED NOTIFICATION FLAGS: 3
RETRY DELAY: 60 NO PROGRESS TIMEOUT: 0 ERROR COUNT: 0
PROXY USAGE: NO_PROXY PROXY LIST: NULL PROXY BYPASS LIST: NULL
DESCRIPTION: VMM LAN Transfer Job
JOB FILES:
42948608 / 42948608 WORKING
https://contoso.com:443/8272f226d8b8cd818406b28de9b6ad0f012b9a55/C:A5cProgramDataA5c
MicrosoftA5cWindowsA5cHyperVA5calbtest1A5cvmguest.iso/E:A5cVirtualMachinesA5calbtes
t1A5cvmguest.iso -> E:\VirtualMachines\albtest1\vmguest.iso

2 - Verify GPO settings
By default, BITS transfers are wide open, not throttled down. Follow the MDSN article and look for the following settings in case they have been changed from default values.
- Any setting with 'Peercache' in the name is not relevant
- The rest of the settings should be set to 'Not Configured' or 'Disabled' to ensure that BITS is not being throttled at all. 'Maximum network bandwidth for BITS background transfers' is most important to be set to 'Not Configured' or 'Disabled'

BITS uses the following Group Policies to configure BITS
http://msdn.microsoft.com/en-us/library/aa362844(VS.85).aspx

The group policies for BITS are located in the registry here…

HKLM\Software\Policies\Microsoft\Windows\BITS

(Note that only those policies that are configured are listed in the registry.)

REMEMBER: Run ‘gpupdate /force’ following any changes from an elevated command prompt

3 - Disable Task Offloading
This work can be done easily on a non-Core server by disabling all properties on the 'Advanced' tab of all NICs. You should then follow the manual directions below to verify the rest of the Offload related settings for the NICs, and to disable this feature in Windows.

If working with a core server you may want to connect via remote registry and make these changes.
This MDSN article lists all values available.

Using Registry Values to Enable and Disable Task Offloading
http://msdn.microsoft.com/en-us/library/aa938424.aspx

All values should be set to '0' EXCEPT for the last one which is related to IPSEC, and thus Kerberos. Leave this one alone. 'HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Ipsec\EnabledOffload'

A. Locate all physical NICs in the registry under:
'HKLM\System\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}'

B. There will be folders with four digits, starting with '0000'. Locate the ones that show the physical NIC names on the right. Look for the 'DriverDesc' value on the right; this should have the name of a NIC such as 'HP NC360T PCIe Gigabit Server Adapter.' For each of these, make the changes below.

C. Disable All Vendor specific Offloading
Set values for any entries below that include the word 'Offload' to '0' (Disabled)
*FlowControl

No description available

*IPChecksumOffloadIPv4
     Describes whether the device enabled or disabled the calculation of IPv4
checksums.
*TCPChecksumOffloadIPv4
     Describes whether the device enabled or disabled the calculation of TCP Checksum
over IPv4 packets.
*TCPChecksumOffloadIPv6
     Describes whether the device enabled or disabled the calculation of TCP checksum
over IPv6 packets.
*UDPChecksumOffloadIPv4
     Describes whether the device enabled or disabled the calculation of UDP Checksum
over IPv4 packets.
*UDPChecksumOffloadIPv6
     Describes whether the device enabled or disabled the calculation of UDP Checksum
over IPv6 packets.
*LsoV1IPv4
     Describes whether the device enabled or disabled the segmentation of large TCP
packets over IPv4 for large send offload version 1 (LSOv1).
*LsoV2IPv4
     Describes whether the device enabled or disabled the segmentation of large TCP
packets over IPv4 for large send offload version 2 (LSOv2).
*LsoV2IPv6
     Describes whether the device enabled or disabled the segmentation of large TCP
packets over IPv6 for large send offload version 2 (LSOv2).
*IPsecOffloadV1IPv4
     Describes whether the device enabled or disabled the calculation of IPsec headers
over IPv4.
*IPsecOffloadV2
     Describes whether the device enabled or disabled IPsec offload version 2
(IPsecOV2). IPsecOV2 provides support for additional crypto-algorithms, IPv6, and
co-existence with large send offload version 2 (LSOv2).
*IPsecOffloadV2IPv4
     Describes whether the device enabled or disabled IPsecOV2 for IPv4 only.
*RSS
     Receive side scaling
*TCPUDPChecksumOffloadIPv4
     Describes whether the device enabled or disabled the calculation of TCP or UDP
checksum over IPv4.
*TCPUDPChecksumOffloadIPv6
     Describes whether the device enabled or disabled the calculation of TCP or UDP
checksum over IPv6.

D. Disable Offloading in Windows.
Use the following registry values to enable or disable task offloading for the TCP/IP protocol:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\TCPIP\Parameters\DisableTaskOffload

Setting this DWORD value to ‘1’ disables all of the task offloads from the TCP/IP transport. Setting this value to zero enables all of the task offloads.

Related Links
Network Issues with Windows Server 2008 RDP and VS/Hyper-V on Dell Servers
http://www.petri.co.il/network-issues-with-windows-server-2008-rdp-on-dell-servers.htm

Technorati Tags: microsoft,scvmm,hyper-v,virtual,virtualization,windows