Jonathan's Virtual Blog : WMI

Are All of My Servers Really Up to Date?

Jonathan — Tue, 21 Apr 2009 20:47:00 GMT

With a single command you can pull a list of all installed updates. Use this to compare patch levels on all systems so you can easily target those not meeting the bar.

We start with simple commands, then build on these to create robust queries with optional CSV and HTML formatting. I think of these commands, which I use on a daily basis, as basics for my Toolkit. If you don’t have a Toolkit I suggest you create one now. Place your often used tools in a folder such as ‘C:\Toolkit’ so that they are always at hand. Grab the first five items under ‘Top 10 Downloads’ from Sysinternals.com to round out your basics.

TIP: Go one step further and sign up for a Live Mesh account with your Windows Live ID. With this you can keep your Toolkit directory, and other directories, synched up across all machines you designate. The folders will be available privately online as well. Further, remote control is built-in for all Mesh enabled systems. Sweet! Clearly remote capabilities are geared more towards workstations than servers, but I use it for both.

And yeah, it’s free. Get started here.

Basic Commands

Warm up by opening an elevated command prompt on your SCVMM server (or any server for that matter). If your command window does not show ‘Administrator:’ as below, you are not running an elevated console.

> Single command to pull updates.
wmic qfe

http://support.microsoft.com/?kbid=963660 VMM2008R2-03 Update                        KB963660
                       NT AUTHORITY\SYSTEM 2/26/2009
http://support.microsoft.com/?kbid=967062 VMM2008R2-03 Update                        KB967062
                       NT AUTHORITY\SYSTEM 2/26/2009

You should see a torrent of information scroll through the command window. This is a list of all QFEs (Hotfixes and Updates) installed, along with a good bit of extraneous info.

> Now let's work on formatting. This is my everyday method.
wmic qfe list brief

Update KB963660 NT AUTHORITY\SYSTEM 2/26/2009
Update KB967062 NT AUTHORITY\SYSTEM 2/26/2009

Now some of the noise is removed and it is easier to read the list of updates.

TIP: Is KBXXXXXX installed on this system?!? I can only remember the last three digits of the update anyway… Easy to check. Just pipe your output to ‘find.’

wmic qfe list brief | find /i "660"

Update KB963660 NT AUTHORITY\SYSTEM 2/26/2009

Apparently it is installed. Nice.

> This is good, but we can do better.
wmic qfe get csname,hotfixid,description

VMM2008R2-03 Update KB963660
VMM2008R2-03 Update KB967062

Now we're getting somewhere. We've selected the header for only those columns we are interested in and have returned a clean list of updates along with the system name.

This much you can keep in your head. You might want to create a batch file for items in the next section.

Advanced Queries

Now that you've waded into the deep end let's perform some real work. These are the commands to place in a batch file for your Toolkit. The steps below assume you have already created a ‘C:\Temp’ directory on your system. CSName is not required for the next two commands; it is implied.

> Create a CSV file from the output.
wmic /output:c:\temp\qfe.csv QFE GET HotFixID,Description /format:csv

The resulting CSV file lists these items, ready to be opened in Excel: Node,Description,HotFixID

> CSV files are not as interesting as HTML, so let's shift output formats.
wmic /output:c:\temp\qfe.html QFE GET HotFixID,Description /format:htable

Now you have a clean multicolored table with results, complete with column headers.

Remote Systems

The only thing left to do is perform this same work on remote machines. First a single machine, then a list of systems. Replace <remotesystem> with your remote system name. Note that we add CSName to pull the remote name.

> List remote system updates.
wmic /node:"<remotesystem>" /output:c:\temp\qfe_remote.html QFE GET CSName,HotFixID,Description /format:htable

If this did not work for you keep in mind that you must be an administrator on the remote machine, and Remote Management must be enabled.

> Finally, let's feed a list of server names and create a single HTML file. Create a file named 'serverlist.txt' with one system per line.
wmic /node:@serverlist.txt /output:c:\temp\qfe_serverlist.html QFE GET CSName,HotFixId,Description /format:htable

That's it. You should have a table similar to this.

Enjoy your new tools. Thanks for reading!

For further reference:

MSDN - WMI Command Line Tools
http://msdn.microsoft.com/en-us/library/aa394531.aspx

Resolve Communication Issues in SCVMM - Part 1

Jonathan — Thu, 15 Jan 2009 22:51:00 GMT

Disable Task Offloading and Update WMI

Communications issues with SCVMM can take on the form of access denied, timeouts, and often as intermittent connectivity issues. The steps in this post are the same ones I use to resolve a good number of customer issues.

Feedback on this article is welcome. If you have technical questions please post them appropriately in the forums:

http://social.technet.microsoft.com/Forums/en-US/category/virtualmachinemanager/

Two types of issues are addressed here

1) Many new settings available to NIC drivers that do not always function as well as would be expected.

2) SCVMM relies heavily on WMI. The updates listed in this article resolve issues with WMI that will allow SCVMM to function properly.

This MDSN article lists all values available.

Using Registry Values to Enable and Disable Task Offloading
http://msdn.microsoft.com/en-us/library/aa938424.aspx

WMI

Install three WMI updates on all Windows 2008 server running Hyper-V (and therefore are SCVMM Hosts)

958124 A wmiprvse.exe process may leak memory when a WMI notification query is used heavily on a Windows Server 2008-based or Windows Vista-based computer

http://support.microsoft.com/default.aspx?scid=kb;EN-US;958124

954563 Memory corruption may occur with the Windows Management Instrumentation (WMI) service on a computer that is running Windows Server 2008 or Windows Vista Service Pack 1

http://support.microsoft.com/default.aspx?scid=kb;EN-US;954563

955805 Certain applications become very slow on a Windows Server 2008-based or Windows Vista SP1-based computer when a certificate with SIA extension is installed

http://support.microsoft.com/default.aspx?scid=kb;EN-US;955805

Network Settings

NOTE

- If working with a core server you may want to connect via remote registry and make these changes.
- All values should be set to '0' EXCEPT for the last one which is related to IPSEC, and thus Kerberos. Leave this one alone ---> 'HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Ipsec\EnabledOffload'

1 - Locate all physical NICs in the registry under:
'HKLM\System\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}'

2 - There will be folders with four digits, starting with '0000'. Locate the ones that show the physical NIC names on the right. Look for the 'DriverDesc' value on the right; this should have the name of a NIC such as 'HP NC360T PCIe Gigabit Server Adapter.' For each of these, make the changes below.

3 - Disable All Vendor specific Offloading
Set values for any entries below that include the word 'Offload' to '0' (Disabled)
*FlowControl

No description available

*IPChecksumOffloadIPv4
     Describes whether the device enabled or disabled the calculation of IPv4
checksums.
*TCPChecksumOffloadIPv4
     Describes whether the device enabled or disabled the calculation of TCP Checksum
over IPv4 packets.
*TCPChecksumOffloadIPv6
     Describes whether the device enabled or disabled the calculation of TCP checksum
over IPv6 packets.
*UDPChecksumOffloadIPv4
     Describes whether the device enabled or disabled the calculation of UDP Checksum
over IPv4 packets.
*UDPChecksumOffloadIPv6
     Describes whether the device enabled or disabled the calculation of UDP Checksum
over IPv6 packets.
*LsoV1IPv4
     Describes whether the device enabled or disabled the segmentation of large TCP
packets over IPv4 for large send offload version 1 (LSOv1).
*LsoV2IPv4
     Describes whether the device enabled or disabled the segmentation of large TCP
packets over IPv4 for large send offload version 2 (LSOv2).
*LsoV2IPv6
     Describes whether the device enabled or disabled the segmentation of large TCP
packets over IPv6 for large send offload version 2 (LSOv2).
*IPsecOffloadV1IPv4
     Describes whether the device enabled or disabled the calculation of IPsec headers
over IPv4.
*IPsecOffloadV2
     Describes whether the device enabled or disabled IPsec offload version 2
(IPsecOV2). IPsecOV2 provides support for additional crypto-algorithms, IPv6, and
co-existence with large send offload version 2 (LSOv2).
*IPsecOffloadV2IPv4
     Describes whether the device enabled or disabled IPsecOV2 for IPv4 only.
*RSS
     Receive side scaling
*TCPUDPChecksumOffloadIPv4
     Describes whether the device enabled or disabled the calculation of TCP or UDP
checksum over IPv4.
*TCPUDPChecksumOffloadIPv6
     Describes whether the device enabled or disabled the calculation of TCP or UDP
checksum over IPv6.

4 - Disable Offloading in Windows.
Use the following registry values to enable or disable task offloading for the TCP/IP protocol:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\TCPIP\Parameters\DisableTaskOffload

Setting this DWORD value to ‘1’ disables all of the task offloads from the TCP/IP transport. Setting this value to zero enables all of the task offloads.

Related Links
Network Issues with Windows Server 2008 RDP and VS/Hyper-V on Dell Servers
http://www.petri.co.il/network-issues-with-windows-server-2008-rdp-on-dell-servers.htm

Updates! Updates! SCVMM, Hyper-V, Cluster

Jonathan — Mon, 29 Dec 2008 22:23:00 GMT

There are many updates for SCVMM and Hyper-V. Some are required just following product install, others are provided to offer greater functionality. Some updates/hotfixes address issues that may constitute a headache for you down the road. I'll list everything I know and provide recommendations. This post will change over time as new updates and issues arise. If you find a hotfix or update that you think should be posted here, please let me know. Include the environment conditions (Windows version, etc) and what the update addresses. Here's to being proactive!

Article Ratings

A – Required by product to operate correctly

B – Resolves known issues

C – Adds Functionality

JJ – Get it. This is on my short list when working with customers.

A – SCVMM – Hyper-V

956589 Description of the Hyper-V update for issues that may occur when you manage the Hyper-V role on the 64-bit editions of Windows Server 2008 by using SCVMM

http://support.microsoft.com/default.aspx?scid=kb;EN-US;956589

A – SCVMM | BITS

956774 A Background Intelligent Transfer Service (BITS) client cannot handle files that have paths that contain the volume GUID in Windows Server 2008 or in Windows Vista

http://support.microsoft.com/default.aspx?scid=kb;EN-US;956774

B – JJ – SCVMM | WMI

958124 A wmiprvse.exe process may leak memory when a WMI notification query is used heavily on a Windows Server 2008-based or Windows Vista-based computer

http://support.microsoft.com/default.aspx?scid=kb;EN-US;958124

B – JJ – SCVMM | WMI

954563 Memory corruption may occur with the Windows Management Instrumentation (WMI) service on a computer that is running Windows Server 2008 or Windows Vista Service Pack 1

http://support.microsoft.com/default.aspx?scid=kb;EN-US;954563

B – JJ – SCVMM - WMI

955805 Certain applications become very slow on a Windows Server 2008-based or Windows Vista SP1-based computer when a certificate with SIA extension is installed

http://support.microsoft.com/default.aspx?scid=kb;EN-US;955805

B – JJ – SCVMM | P2V

959596 Description of the System Center Virtual Machine Manager 2008 update to address physical to virtual (P2V) issues

http://support.microsoft.com/default.aspx?scid=kb;EN-US;959596

B – JJ – Hyper-V | VSS

956697 Windows Server 2008 Hyper-V VSS writer is not used during a backup job because of corrupted or invalid virtual machine configuration files

http://support.microsoft.com/default.aspx?scid=kb;EN-US;956697

B – Hyper-V | VSS

959978 Error message when you back up a Windows Server 2003-based virtual machine on a Windows Server 2008 Hyper-V-based computer: "GetWriterStatus FAILED for Selected writer [Microsoft Hyper-V VSS Writer], writer is in state [9] [VSS_WS_FAILED_AT_FREEZE]"

http://support.microsoft.com/default.aspx?scid=kb;EN-US;959978

B – Hyper-V |SCVMM

957967 Stop error message on a Windows Server 2008-based computer that has the Hyper-V role installed: "STOP 0x0000001A"

http://support.microsoft.com/default.aspx?scid=kb;EN-US;957967

B – Cluster | Hyper-V

957311 Recommended hotfixes for Windows Server 2008-based server clusters

http://support.microsoft.com/default.aspx?scid=kb;EN-US;957311

B – JJ - Hyper-V | Drive Letters

958184 Virtual machine backup operations fail in Windows Server 2008 when Hyper-V virtual machine files are saved on a volume that is mounted on a failover cluster by using a volume GUID

http://support.microsoft.com/default.aspx?scid=kb;EN-US;958184

B – Hyper-V | Cluster

958065 You cannot configure a Hyper-V virtual machine by using Windows Server 2008 Failover Clustering when the virtual machine uses a storage device that is managed by a third-party clustered file system or a third-party replication solution

http://support.microsoft.com/default.aspx?scid=kb;EN-US;958065

C – Hyper-V | Cluster

951308 Increased functionality and virtual machine control in the Windows Server 2008 Failover Cluster Management console for the Hyper-V role

http://support.microsoft.com/default.aspx?scid=kb;EN-US;951308

Technorati Tags: microsoft,scvmm,hyper-v,virtual,virtualization,windows

WMI Errors and SCVMM – If It’s Not Broken, Update It!

Jonathan — Mon, 29 Dec 2008 12:02:00 GMT

There are many errors that may appear, based on events in SCVMM, to be WMI related. An understandable response would be to troubleshoot WMI. On the other hand, many ‘Access Denied’ errors, while pointing to WMI in no way, are actually the result of WMI issues. Below are a number of error examples that in one way or another are related to WMI. The best way to deal with these errors is divide and conquer. First, determine if WMI itself is responding correctly. By this I mean that we test WMI functionality alone, not in combination with SCVMM which has its own detached namespace. Second, with no fundamental problems found with WMI due to NIC teaming or otherwise, there are three updates that may very well resolve your issue. These should be installed on all Windows 2008 servers that hold SCVMM components. The updates are at the bottom of this post. Good luck!

_____ERRORS_____

SCVMM Admin Console typical errors for Hosts
'Connection Status: Not responding'
'Virtualization service status: unknown"

WMI typical errors
Example #1
Log Name: Application
Source: Microsoft-Windows-WMI
Event ID: 10
Task Category: none
Description:
Event filter with query "select * from __instancemodificationevent within 30 where
targetinstance isa 'Win32_PerfFormattedData_PerfOS_Processor' and
targetinstance.PercentProcessorTime > 99 and targetinstance.Name != '_Total'" could
not be reactivated in namespace "//./root/CIMV2" because of error 0x80041010.
Events cannot be delivered through this filter until the problem is corrected.

Example #2
Log Name: Application
Source: Application Error
Event ID: 1000
Task Category: (100)
Description:
Faulting application wmiprvse.exe, version 6.0.6001.18000, time stamp 0x4791950f,
faulting module ntdll.dll, version 6.0.6001.18000, time stamp 0x4791adec, exception
code 0xc0000005, fault offset 0x000000000003e39c, process id 0x944, application
start time 0x01c95ba1faa603ce.

Jobs list errors in SCVMM Admin Console
Example #1

Error (2911)
Insufficient resources are available to complete this operation on the
server.contoso.com server.
(Not enough storage is available to complete this operation (0x8007000E))
Recommended Action
Ensure that the virtual machine host has sufficient memory and disk space to
perform this action. Try the operation again.

Example #2

Warning (10803)
Unable to refresh \\server.contoso.com\ISOS$\WinPE_X86_with_HyperV_ICs.iso because
the file is in use by another process.
Recommended Action
Wait for the next automatic library refresh, or manually refresh the library share
after the process completes.

Example #3
Error (2912)
An internal error has occurred trying to contact an agent on the server.contoso.com
server.
(No more threads can be created in the system (0x800700A4))
Recommended Action
Ensure the agent is installed and running. Ensure the WS-Management service is
installed and running, then restart the agent.

Example #4
Error (2912)
An internal error has occurred trying to contact an agent on the
ephypv01.constrution.enet server. (The paging file is too small for this operation
to complete (0x800705AF)

_____TEST_____
Verify that various WMI namespaces can be connected to, then update WMI on all involved Windows 2008 machines

Using wbemtest
- Click the 'Start' button
- Start> Run> wbemtest
     - Click 'Connect' and enter 'root\cimv2' then click 'Connect' again. This should not prompt with an error
     - Click 'Connect' and enter 'root\default' then click 'Connect' again. This should not prompt with an error
On a Hyper-V server, this should also work
     - Click 'Connect' and enter 'root\virtualization' then click 'Connect' again. This should not prompt with an error.

On an SCVMM Server (Host) this should also work
- Click 'Connect' and enter 'root\scvmm' then click 'Connect' again. This should not prompt with an error
- Close 'wbemtest'

Using command line
- From an elevated command prompt type 'wmic nic' and press Enter
- This should return information, not an error. If there is an error, and all other tests work, this is likely due to NIC TEAMING. Uninstall all NIC Teaming software, reboot and try again
- From an elevated command prompt type 'wmic diskdrive list brief' and press Enter
- This should return information, not an error

_____UPDATE_____

Install WMI Updates and Hotfixes
Each requires a reboot

954563 Memory corruption may occur with the Windows Management Instrumentation (WMI) service on a computer that is running Windows Server 2008 or Windows Vista Service Pack 1
http://support.microsoft.com/default.aspx?scid=kb;EN-US;954563

958124 A wmiprvse.exe process may leak memory when a WMI notification query is used heavily on a Windows Server 2008-based or Windows Vista-based computer
http://support.microsoft.com/default.aspx?scid=kb;EN-US;958124

955805 Certain applications become very slow on a Windows Server 2008-based or Windows Vista SP1-based computer when a certificate with SIA extension is installed

http://support.microsoft.com/kb/955805

Technorati Tags: microsoft,scvmm,hyper-v,virtual,virtualization,windows

Troubleshooting and Data Collection During Offline P2V

Jonathan — Mon, 29 Dec 2008 11:36:00 GMT

When the P2V process fails there can be a number of issues to identify. A number of situations are discussed below. The machine being virtualized is referred to as the Source machine. The location where the virtualized machine is being copied is the Destination Host.

Use this article as your main reference for all P2V issues
P2V: Converting Physical Computers to Virtual Machines in VMM
http://technet.microsoft.com/en-us/library/cc764232.aspx

Requirements for the Source Machine
To perform a P2V conversion, your source computer:
- Must have at least 512 MB of RAM.
- Must have an Advanced Configuration and Power Interface (ACPI) BIOS - Vista WinPE will not install on a non-ACPI BIOS.
- Must be accessible by VMM and by the host computer.
- Cannot be in a perimeter network (also known as a DMZ, demilitarized zone, and screened subnet) where the firewalls or IPsec settings prohibit communication.
- Must contain one of the following supported operating systems:
- Windows Server 2008 (32-bit)
- Windows Server 2008 (64-bit)
- Windows Server 2003 (32-bit) SP1 or later
- Windows Server 2003 (64-bit) SP1 or later
- Windows 2000 Server SP4 or later (Offline P2V only)
- Windows 2000 Advanced Server SP4 or later (Offline P2V only)
- Windows XP Professional (32-bit) SP2 or later
- Windows XP Professional (64-bit) SP2 or later
- Windows Vista (32-bit) SP1 or later
- Windows Vista (64-bit) SP1 or later

Errors and Resolutions
- Third party applications can lead to failures
- Disable all antivirus using MSCONFIG and reboot
- Disable all monitoring software such as Insight Manager (optional)
- Disable IIS on the source machine if running

Network Related - Source machine
Make sure that when rebooted into WinPE there is an IP address. There is a command prompt on the console. Type 'ipconfig /all' and verify with the customer that all IP settings are correct and match that of the SCVMM Server and Destination Host

If the IP starts with 169.254.x.x this is an auto-configed IP. This means there is no DHCP serve to provide an IP address or there are problems with the NIC drivers. It is preferable that the source and destination host are on the same subnet. If not test all routing by pinging…
     - Ping the SCVMM and Host SCVMM servers by DNS name, not IP. If they do not resolve, correct DNS
     - Ping the IP address of the default gateway and DNS servers. If they do not reply, correct IP settings
     - If IP information needs to be set manually in WinPE, at a command prompt type:
‘netsh interface ipv4 set address "Local Area Connection" static 10.241.0.7 255.255.0.0    10.241.0.2    1’
     Where:
‘netsh interface ipv4 set address "Local Area Connection" static %ipaddr% %subnet% %dns server% 1’

Network Related - Source and SCVMM machines
On Windows 2003 machines follow the 'Workaround' in this article (disregard the fact that it is a SQL KB. It is correct). This will disable TCP Offloading in Windows 2003 SP2, which can lead to a number of networking issues.

     942861 Error message when an application connects to SQL Server on a server that is running Windows Server 2003:
     "General Network error," "Communication link failure," or "A transport-level error"
     http://support.microsoft.com/default.aspx?scid=kb;EN-US;942861

WMI and 'Access Denied' errors
Follow the article on this site ‘WMI Errors and SCVMM’

During 'Scan System' operation
Example Error
VMM does not have appropriate permissions to access the resource on the <destination_computer> server. Ensure that Virtual Machine Manager has the appropriate rights to perform this action.
ID: 2910
Details: Access is denied (0x80070005)

Resolution #1
- Make sure the computer object for the SCVMM server is in the Administrators group on the source machine
- Make sure the account specified in the SCVMM Admin Console is a member of the Administrators group on the source machine.
- SCVMM 2008 - If the machine is in a workgroup, make sure the Domain specified along with the Account and Password is actually the name of the source machine, not the domain of the SCVMM server

Resolution #2
Ensure that the following services are not set to 'Disabled,' and that you can start them successfully on the source machine
     - WMI
     - Background Intelligent Transfer Service (BITS)
     - HTTP SSL
     - RPC

Resolution #3
Enable auditing of object access to determine if the account specified in SCVMM or the SCVMM computer object are failing to access the source machine as required. Look for 'Failure' in the security log once configured and tested.

- Follow the first section of the article below 'For the local computer'
     325898 How to set up and manage operation-based auditing for Windows Server 2003, Enterprise Edition
     http://support.microsoft.com/default.aspx?scid=kb;EN-US;325898

- Record any settings under 'Audit Policy' that are already configured
- Change configure all Policies to record Success and Failure
- If any Policies have an icon of gray servers instead of blue 0's and 1's you will not be able to make this change and must involve the customer's Group Policy administrator
- Back at a command prompt type 'gpupdate /force'
- Try the scan again, then check the Security log for ’Failure’ audits
- Set all Policies back to the state you recorded them as before making changes

Resolution #4
Make sure the DCOM (DCOMLaunch) service is set to Automatic and running. (Thanks to Pramod for this tip)

Non-ACPI HAL
If the source machine has a non ACPI HAL P2V cannot be performed.

P2V: Converting Physical Computers to Virtual Machines in VMM
http://technet.microsoft.com/en-us/library/cc764232.aspx

309283 HAL options after Windows XP or Windows Server 2003 Setup
http://support.microsoft.com/default.aspx?scid=kb;EN-US;309283

249694 How to move a Windows installation to different hardware
http://support.microsoft.com/default.aspx?scid=kb;EN-US;249694

Technorati Tags: microsoft,scvmm,hyper-v,virtual,virtualization,windows,P2V