Cheng's Random Thoughts on System Management : Debugging

VMM Configuration Analyzer 2008 is now available for download!

Cheng Wei — Fri, 05 Dec 2008 05:06:00 GMT

Greetings folks,

Today, I've got a guest poster, Carmen Asher, one of my colleagues in VMM team. Carmen is the program manager responsible for SCVMM Configuration Analyzer 2008. Her post below provides you with all the information you need to know on: what VMMCA is all about, where to download it, how to use it, and she is nice enough to even prepare a section of frequently asked questions. J

Take advantage and enjoy this free and powerful tool for VMM 2008!

Cheng

------

Hi, I am Carmen Asher, Program Manager on System Center Virtual Machine Manager. I am using a spot on Cheng’s blog to announce the Public Release to Web of the Virtual Machine Manager Configuration Analyzer 2008.

Overview

Customers have come to rely upon, and appreciate, the added value that VMMCA tool provides, so getting it out within weeks of RTM is critical for customers to assist in helping plan implementation and then also provide value-add during the ongoing operational phase of SCVMM 2008.

Description

The VMMCA is a diagnostic tool you can use to evaluate important configuration settings for computers that either are serving or might serve VMM roles or other VMM functions. The VMMCA scans the hardware and software configurations of the computers you specify, evaluates them against a set of predefined rules, and then provides you with error messages and warnings for any configurations that are not optimal for the VMM role or other VMM function that you have specified for the computer.

You can use VMMCA before installing VMM, to help you evaluate the configuration of computers that you might use for the following VMM roles:

· VMM server

· VMM Administrator Console

· VMM Self-Service Portal

You can also use VMMCA after installing VMM, to evaluate the configurations of computers that you are now using or might use for the VMM roles listed above, as well as computers that you are using or might use for any of the following VMM functions:

· Windows Server–based host

· VMware Virtual Center Server

· P2V source computer

· Operations Manager agent

System Requirements:

Before you install the VMMCA, you must download and install the 64-bit version of Microsoft Baseline Configuration Analyzer (MBCA). To download the MBCASetup64.msi file, go to http://go.microsoft.com/fwlink/?LinkID=97952.

Note

The system requirements for the MBCA indicate that only Windows Server 2003 is supported. However, the VMMCA and the MBCA have been tested and are supported on Windows Server 2008 64-bit.

You must install and run the VMMCA on the computer that either currently is or will become your VMM 2008 server. To review the system requirements for the VMM 2008 server, go to http://go.microsoft.com/fwlink/?LinkId=133516.

Download VMMCA Now

Common Usage Scenarios

The list of scenarios are not comprehensive, but will provide you with some common usage scenarios to provide you with an understanding of when to use VMMCA 2008.

Scenario 1 – I have a Windows Host that is in a not responding state. Use VMMCA 2008 to check the host for commonly known issues.

Scenario 2 – A Windows host shows a status of “Needs Attention”. Use VMMCA 2008 to check the this host for commonly known issues.

Scenario 3 – I am using SCVMM 2008 to manage a VMware environment, but I am unable to create new virtual machines on the VMware environment. Use VMMCA 2008 to evaluate the Virtual Center environment for commonly known issues.

Scenario 4 – I am using the integration between SCVMM 2008 and Operations Manager, but I am unable to see a particular VM or Host in the integrated mapping view. Use VMMCA 2008 to check the host or virtual machine for an operations manager agent.

Scenario 5 – I am using SCVMM 2008 to do P2V conversions and the job is failing. Use VMMCA 2008 to check the P2V source for commonly known issues.

Scenario 6 - I tried to create a Virtual Machine and it failed during customization and installation of virtual guest services. Use VMMCA 2008 on the Windows Host to find commonly known issues.

Frequently Asked Questions

Question: Do I have to install MBCA 1.0 and VMMCA 2008?

Answer: Yes, VMMCA 2008 uses the MBCA 1.0 64 bit engine that enables the rule checks for VMMCA 2008.

Question: Where can I install VMMCA 2008?

Answer: VMMCA 2008 is installed on a Windows 2008 64 bit server that is or will become the VMM Server.

Question: MBCA 1.0 64 bit system requirements state that it is only installable on Windows 2003 platform. Can I install it on Windows 2008 64 bit.

Answer: The SCVMM 2008 has tested the MBCA 1.0 on the Windows 2008 64 bit platform for all of the rule sets defined in VMMCA 2008.

Question: Does VMMCA 2008 perform all pre-requisite checks for SCVMM 2008?

Answer: No, the VMMCA does not duplicate or replace the prerequisite checks performed during the setup of VMM 2008 components.

Question: Can I run a check against a individual ESX hosts?

Answer: No, VMware checks are run by selecting the “other computers” check box and then from the drop menu selecting VMware VirtualCenter. Add the VirtualCenter name and run the scan. VMMCA 2008 will access the managed ESX hosts by using information in VMM 2008.

All about refreshers ...

Cheng Wei — Thu, 02 Oct 2008 02:19:00 GMT

If you have been using VMM for a while, you’ve probably figured out that we use various “refreshers” to pull data from the hosts and VMs that we manage. When deploying VMM in various sized environments, depending on the perf/scale requirements, you might need to tweak the refresher schedule to achieve the best performance.

So, to help you understand what’s happening “under the hood”, here are the refreshers we use in SCVMM 2008 product. Please note, that not all refreshers are customizable.

Host Refresher: Runs every 30mins on every host (with time lags) and updates host properties. It also updates disk/SAN and host network (NIC / virtual switch) information. It does not check any VM related properties or perf counters on hosts. Can be manually triggered on the UI or through cmdlet.

VM Refreshers: Two types

VM Heavy Refresher: Runs every 30mins on every Host and also runs whenever a VM is clicked on (or when you call refresh-vm cmdlet). It updates all VM properties, resource pools, clustering information for this VM and snapshots. But it does not update VM performance counter info.

VM Light Refresher: Runs every 2mins on every Host that is currently in the VMM database. This refresher does the following:

- Checks the host state (to see if the agent is running or the agent is down) and virtualization software status

- Syncs the state of a VM

- Marks VMs as missing

- Imports new VMs created outside of VMM (When VMM detects a new VM created on the virtualization platform, we import it into VMM and kick off the heavy refresher for the new VMs)

Cluster Refresher: Runs every 30mins and refreshes all cluster properties (including bringing in newly added nodes or taking our removed nodes). Can be manually triggered on the UI or through cmdlet.

Library Refresher: Runs on user configurable schedule (increment by 1 hour interval and can be turned off completely) and updates the library shares info and library objects. Can be manually triggered on the UI or through cmdlet.

Perf Refresher: Runs every 9 minutes on a host or whenever there is any state changing operation on the VM (i.e. start/stop/save/etc.). It collects perf counter information of both the host and all the VM’s on a given host.

VirtualCenter Refresher: Runs every 30mins and refreshes VC properties, ESX hosts and resource pools that are managed by this VC. Can be manually triggered on the UI or through cmdlet.

User Role Refresher: Runs every 30mins and updates user role properties.

PRO Tip Refresher: Runs every 30secs. It looks for PRO specific alerts in OpsMgr and reconciles the PRO tips in our DB against the data that is brought back from OpsMgr. This refresher cannot be manually triggered.

Thanks for reading and hope this is useful,

Cheng

How to collect WinRM traces

Cheng Wei — Fri, 29 Aug 2008 02:59:00 GMT

Since “How to collect SCVMM traces” and “What to do when I have hosts in Needs Attention state?” are published, we've seen requests on how to further debug WinRM/WMI issues that lead to VMM AddHosts failures.

It’s particularly useful, when you run into a SCVMM error that is rooted from a WinRM error code with a generic error message. So, to make your debugging effort easier, here is the process to collect WinRM traces, which hopefully provide more clues on the root cause of the WinRM failure on the remote host with help from your Microsoft support:

1) Download the Windows SDK toolkit:

1. For Windows 2008 servers:

1. If the server you’re trying to collect WinRM traces from has Internet access, download tracelog from: http://www.microsoft.com/downloads/details.aspx?FamilyId=E6E1C3DF-A74F-4207-8586-711EBE331CDC&displaylang=en;

2. if not, you can download the iso image (about 1.4G) of the Windows SDK from here and burn it into a DVD: http://www.microsoft.com/downloads/details.aspx?FamilyId=F26B1AA4-741A-433A-9BE5-FA919850BDBF&displaylang=en

2. For Windows 2003 or client (XP / Vista) systems:

1. Download it from if you have Internet access: http://www.microsoft.com/downloads/details.aspx?FamilyID=4377f86d-c913-4b5c-b87e-ef72e5b4e065&displaylang=en&Hash=voZ0UJrspn5qT6W4kcu%2fOMSjQf75oB2WCTbQ262yAcHxnRmEWyOw6qqQlesatQ8YZBNHBRZQxyYohHYvUNyNPA%3d%3d

2. Download the ISO file from: http://www.microsoft.com/downloads/details.aspx?FamilyId=F26B1AA4-741A-433A-9BE5-FA919850BDBF&displaylang=en

2) Install The Windows SDK to one of the folders that are in you default path.

3) Save the following text into a starttracing.cmd file to the folder where you install Windows SDK (in particular tracelog.exe).

@echo off

@echo on

REM Usage:

REM this.cmd - turns all tracing on

REM this.cmd 0xHHHH - turns tracing w/ for specific bits

REM see bits in WPP_DEFINE_CONTROL_GUID for more details

REM bits a counted from first entry (0th bit) to last entry (Nth bit)

REM

REM E.g.: this.cmd 0x1C00 will turn on tracing for automation component

REM

@echo off

set FLAGS=0x1

if NOT {%1}=={} (

set FLAGS=%1

)

@echo FLAGS=%FLAGS%

@echo ------------------------------------------------------------------------------------------------

@echo attempting to stop and re-start Wmxlog logger for logging

@echo ------------------------------------------------------------------------------------------------

tracelog.exe -stop Wmxlog

tracelog.exe -start Wmxlog -f wmxlog.etl -guid tracectl.txt -ft 1 -flag %FLAGS%

@echo ------------------------------------------------------------------------------------------------

@echo run your application now and then use displaytrace.cmd to see the traces logged

@echo ------------------------------------------------------------------------------------------------

goto End

:CtlNotExist

@echo tracectl.txt is missing

goto End

:NtTreeNotDefined

@echo variable _NTTREE is not defined

goto End

:TraceFormatDirNotExist

@echo directory %_NTTREE%\Symbols.pri\TraceFormat (_NTTREE\Symbols.pri\TraceFormat) does not exist

goto End

:End

4) Save the following text into a stoptracing.cmd file to the folder where you put tracelog.exe.

@echo off

@echo ------------------------------------------------------------------------------------------------

@echo stop Wmxlog logger

@echo ------------------------------------------------------------------------------------------------

tracelog.exe -stop Wmxlog

5) Save the following text into a tracectl.txt file to the folder where you put tracelog.exe.

04c6e16d-b99f-4a3a-9b3e-b8325bbc781e WinRM

c0a36be8-a515-4cfa-b2b6-2676366efff7 WinrsMgr

f1cab2c0-8beb-4fa2-90e1-8f17e0acdd5d WinrsExe

03992646-3dfe-4477-80e3-85936ace7abb WinrsCmd

6) Run “starttracing.cmd”.

7) Repro your steps that you see the WinRM failure.

8) Run “stoptracing.cmd”. The resulting wmxlog.etl is the trace for WinRM.

Please note that the trace is unreadable since it's binary data. It needs to be sent to your Microsoft support contact to render and explain what has happened in the recorded trace.

Hope this helps and thanks for reading,

Cheng

How to collect SCVMM traces

Cheng Wei — Thu, 08 May 2008 20:31:00 GMT

In order to diagnose a failure scenario, it is normally required to reproduce the issue and collect traces at the same time.

Here are the instructions on how to collect the traces:

From what computer should I collect the trace?

If it’s a console crash issue,

please collect the traces from both the computer where you run admin console, and your VMM Server.

If it’s an “Add Hosts” issue,

please collect the traces from the VMM server;

If it’s a host status (Needs Attention) or VM issue,

please collect the traces from both the VMM server and the host in question.

If it's self-service portal issue,

please collect the traces from the Web server and the VMM server

What are the steps to collect traces?

Install DebugView from http://www.microsoft.com/technet/sysinternals/utilities/debugview.mspx on your VMM server, your host in question and your Web server (if it's a self-service portal issue).
Save the following code into a text file and name it as "odsflags.cmd":

@echo off
echo ODS control flags - only trace with set flags will go to ODS

if (%1)==() goto :HELP
if (%1)==(-?) goto :HELP
if (%1)==(/?) goto :HELP

echo Setting flag to %1...
reg ADD "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Tracing\Microsoft\Carmine" /v ODSFLAGS /t REG_DWORD /d %1 /f
echo Done.
goto :EXIT

:HELP
echo Usage: odsflags [flag], where flag is
echo TRACE_ERROR = 0x2,
echo TRACE_DBG_NORMAL = 0x4,
echo TRACE_DBG_VERBOSE = 0x8,
echo TRACE_PERF = 0x10,
echo TRACE_TEST_INFO = 0x20,
echo TRACE_TEST_WARNING = 0x40,
echo TRACE_TEST_ERROR = 0x80,

:EXIT

Save the following code into a text file and name it as "odson.reg":

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Tracing\Microsoft\Carmine]
"ODS"=dword:00000001

Save the following code into a text file and name it as "odsoff.reg":

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Tracing\Microsoft\Carmine]
"ODS"=dword:00000000

Copy the above three files onto your VMM server, your host in question and your Web server (if it's a self-service portal issue).
In a command window on the machine that you want to capture VMM tracing, run “odson.reg” and “odsflags.cmd 255”. (If you need to collect traces for both VMM Server and the host or the Web server, make sure to run these commands on all computers.)
Open DebugView and run it as administrator, make sure that in its Capture menu, you have both "Capture Win32" and "Capture Global Win32" checked. You should be able to see tracing from the VMM components showing up in the DebugView. (If you need to collect traces for both VMM Server and the host, make sure to do these steps on all computers.)
Restart vmmservice on VMM server with “net stop vmmservice” and “net start vmmservice”.
Restart the agent service on the host with “net stop vmmagent” and “net start vmmagent”.
Restart the IIS service on the Web server with "iisreset".
Reproduce the issue that you found.
Save the output from the DebugView to a text file and email it to the people who can help you diagnose the issue.
Don't forget to turn off the tracing after you are done collecting by running "odsoff.reg" on the machine.

Thanks for reading!

Cheng

Unable to add host or hosts in "Needs Attention" state (SCVMM 2008 Beta)?

Cheng Wei — Fri, 02 May 2008 20:53:00 GMT

These are probably two of the most frequently asked questions in the Technet SCVMM forum.

I'm unable to add host, what did I do wrong?
What do I need to do when I have host in "Needs Attention" state or my "Connection Status" is "Not Responding"?

The first thing I'd ask people to do is to check the FAQ published on Connect site along with the beta bits, which covers the known issues and limitation for this beta release. One of the things we documented there is that a USB drive could cause your host be in this "Needs Attention" state. We're fixing the issue for our RTM release.

Secondly, I'd recommend you to check the troubleshooting note published for our VMM 2007 release, which covers a lot of commonly seen and known issues for both add host (or agent install) scenarios and hosts in "Needs Attention" state. The most commonly seen causes of failure are:

Host Hyper-V service is not working properly. The causes include:

Host computer's hardware is not virtualization capable (check here for what Hyper-V requires for hardware-assisted virtualization).
Host computer's hardware virtualization is not enabled. What we found is that most Intel chip based computer's VT is not enabled in factory default, AMD-chip-based computer's is enabled by default.
In either case, you might need to check the BIOS to confirm.

Firewall rules on the host or in the network blocks the initial DCOM communication, or HTTP/HTTPS communications between VMM server and the to-be-managed host.

Turning off the firewall during the initial deployment may help identify the causes of AddHosts failure.
Please note, once the agent is deployed, SCVMM server does not use DCOM to communicate with the hosts. Instead, we use HTTP(for WS-MAN)/HTTPS(for BITS) to communicate with the hosts. As part of the agent install, we add firewall exceptions to allow HTTP and HTTPS traffic to the host. Therefore, once the host agent is deployed, as long as your HTTP and HTTPS ports are open, VMM server should be able to connect to your hosts.

SCVMM user and group was removed for DMZ host or untrusted domain joined host.

During agent deployment, SCVMM creates a group SCVMM and a user starting with string "SCVMM". And the user "SCVMMxxxxx" is added to the local Administrators group and the Virtual Machine Manager Servers group.
We've seen cases where the user and group created by SCVMM are removed by group policy automatically, or removed manually by mistake. Without this user, VMM server will not be able to connect to the host.
Please note, in trusted domain joined host scenario, we do not create local user. We use VMM Server machine account to authenticate and communicate with the host.

Incorrect version of Hyper-V

For this SCVMM 2008 Beta release, we support Hyper-V RC0 release (KB949219).

Lastly, here are some additional testing you could do on your host to unveil issues with services that VMM is dependent of:

To check integrity of or repair your host WMI repository,

run "winmgmt /salvagerepository".

To check your WMI virtualization store,

start wbemtest by using a local admin account of the host machine and
connect to \\<host>\root\virtualization
Try query "select * from msvm_ComputerSystem" and see if it succeeds.

To check your host's WMI namespace security settings,

Go to Server Manager, click on Configuration then right-click on WMI Control and choose Properties
Click on Security tab, choose SCVMM and check its Security Settings.

Make sure that both Administrators group and Virtual Machine Manager Servers group have the full rights on the namespace.

To check your WnRM state,

run "winrm qc".

To check if the WinRM listener is running,

run "winrm enum winrm/config/listener", or
run "winrm get winrm/config" to get the entire WinRM config.

To check if your VMM agent is responding to winrm calls,

run "winrm invoke GetVersion wmi/root/scvmm/AgentManagement -r:http://<host> @{}".

To manually restart VMM server service:

run "net stop vmmservice", and run "net start vmmservice".

To manually restart VMM host agent service:

run "net stop vmmagent", and run "net start vmmagent".

To manually restart Hyper-V service:

run "net stop vmms", and run "net start vmms".

Debugging

If above resources cannot help you find the root cause, try collect debugging traces by following the instructions posted here, study the trace logs and look for exceptions from the traces to get a clue on where things might have gone wrong.

Hope this helps and thanks for reading,

Cheng