Cheng's Random Thoughts on System Management : Agent Deployment and Host Configuration

Got a "Bad Gateway"?

Cheng Wei — Tue, 14 Apr 2009 19:41:00 GMT

Greetings, folks!

Hope you had a great Easter on last Sunday, if you celebrate that holiday! My kids certainly enjoyed hunting eggs in a typical Seattle drizzling Saturday morning. J

This blog entry is for folks who use or plan to use VMM to manage VMware ESX hosts.

If this is your first time adding your ESX host to VMM, and your ESX hosts are added into VMM by using DNS name (instead of IP address), and your network environment with Internet proxy is not made aware of VMware host DNS names, you may see error similar to below:

Error (2891)

VMM cannot complete the …. action on the vmm-esx1.scvmm.contoso.com server because of the error: The remote server returned an error: (502) Bad Gateway.

(Unknown error (0x194))

This issue may occur when:

· To turn your ESX host from “OK(Limited)” state to “OK” state (to get your ESX host in full management mode), you are providing root user credentials and attempting to retrieve the certificate and SSH key from the ESX host to approve.

· You may also run into the same error when attempting to perform a live migration (VMotion) for a VM hosted by an ESX host from VMM admin console.

The cause of this issue is due to the DNS resolution (or proxy configuration) on your VMM server:

· VMM uses web services to communicate to VMware Virtual Center and ESX hosts.

· Proxies are generally configured to permit pass-through of local DNS names.

· In some cases, when the proxy is not configured for a DNS name, the web service connection may fail.

Following are some simple options to resolve this issue:

· SOLUTION 1: Include VMWare host DNS names in the internet proxy configuration.

· SOLUTION 2: Add dns hosts to the proxy configuration of Internet Explorer

· SOLUTION 3: Create a file "vmmservice.exe.config" in the same location as vmmservice.exe (by default, it’s installed under this directory: %ProgramFiles%\Program Files\Microsoft System Center Virtual Machine Manager 2008\bin\) with contents similar to the following. Note: you will need to bounce the VMM server service (restart “Virtual Machine Manager” service from services.mcs) for it to pick up the new proxy configuration).

<system.net>

</bypasslist>

</defaultProxy>

</system.net>

</configuration>

On the extreme side, you could also completely turn off the proxy on your VMM server by applying the following config file:

<system.net>

</defaultProxy>

</system.net>

</configuration>

Hope this helps and thanks for reading!

Cheers,

Cheng

When should I consider running my VMM Service by using a domain account?

Cheng Wei — Fri, 20 Feb 2009 05:19:00 GMT

Greetings folks!

Hope you all had a good Valentine weekend last week!

If you noticed, during your VMM Server installation, there is an option of allowing you to select a different account to run VMM Server service (VMMService) from the default computer account.

“When should I consider using this non-default option?”, you may ask.

You may have other reasons / policies to run VMMService by using a domain account. However, based on some of our recent CSS reports, choosing your own domain account to run VMMService should be a preferred option for customers who are running a more restrictive AD environment. Here is why:

· With default install option, VMMService is run under the VMM Server local system / computer account.

· When adding trusted domain-joined VM hosts (whose domain has two-way trusts with the domain VMM Server is in), VMM Server adds its computer account (the account it uses to run VMMService) into the local administrator group of the target VM hosts, as part of the Add-VMHost process.

· In a more restrictive AD environment, we find it common for customers to have a “Restricted Groups” group policy that disallows machine accounts to be part of the local administrators group. Hence, when the GP is in effect, the machine account will be removed by this GP.

· And when this happens, the affected VM Hosts will show up in VMM console as “Needs Attention” (and the agent status will be “Not Responding”), since the VMM Server will no longer be able to authenticate with the hosts. Here is error message that you will see from the failed host refresher job (BTW, we’ll be updating this error message in our vNext):

Error (2927)

A Hardware Management error has occurred trying to contact server servername.domainname.com.

(Unknown error (0x80338104))

Recommended Action

Check that WinRM is installed and running on server servername.domainname.com. For more information use the command "winrm helpmsg hresult".

When users get into this situation, there are a few options they can do to fix this issue:

1. Check with your IT security group and see if it’s possible to disable the “Restricted Groups” group policy in your Active Directory environment; or

2. Check with your IT security group and see if it’s possible to modify the group policy to allow the VMM machine account in the Local Administrators group; or

3. Check to see if it’s possible to move the VMM Server machine account to its own organizational unit (OU) and block the group policy from being applied to that OU; or

4. If making changes to your group policy (or negotiating with your IT security group J) is next-to-impossible, the only option left is to reinstall the VMM server and choose the option to run the VMM service by using a domain account with admin privilege on your VMM Server computer (in this case, you will need to remove and re-add all your VM hosts, or choose to reinstall your VMM Server without retaining data).

Hence, I highly recommend users to evaluate your IT security (AD) policies before deploying your VMM server into production environment, as those factors do directly affect how VMM performs operations within that environment. And, if you do have a more restrictive AD environment, I suggest you to use a domain account to run VMM Server service. Also, if you have a disjoint namespace environment, it's also recommended to use a domain account to run your VMM Server service.

Before I close on this subject, there is one restriction that I think folks should be aware when using a domain account to run your VMM Server service:

· Users cannot use the same domain user account to add or remove hosts.

o Say, you configured VMM Server to use account “foo\bar” to run VMM Server service.

o And it also happens to be part of the local admin group for a host “MyNewHost” that you want to add to VMM.

o When we go through AddHost wizard (or through our cmdlet), you will be asked about a credential with admin privilege for us to install agent.

o At this point, it’s disallowed to use the same user account “foo\bar” to add the host. And yes, we actually block you from doing such operation.

o The same is true for host removal, it’s disallowed to use the same account “foo\bar” to remove the host.

· Why do we not allow this?

o During host addition, we add the service account to the local admin group on the host. When removing the host, we need to remove the account from the local admin group.

§ If we remove the account first, we won't be able to talk to the agent.

§ If we remove the agent first, we leave the account behind.

§ Thus, users need to use a different account for host removal.

o During host addition, we add the service account to the local admin group on the host.

§ In case of failure during the agent install process as part of the AddHost task, we need to be able to roll back and successfully remove the agent.

§ To do that, we need the same requirement.

· So, the proper process is that if you use “foo\bar” to run your VMM Server service, you will need to use a different account with admin privilege to add or remove your host.

Hope this helps and thanks for reading!

Thanks,

Cheng

Managing VM hosts in a disjoint namespace environment

Cheng Wei — Wed, 28 Jan 2009 21:43:00 GMT

Greetings folks and Happy Chinese New Year to you if you celebrate that cultural tradition!

Today, I wanted to talk about a topic that many of our TAP customers have requested support for during our VMM 2008 release development phase. We’ve seen customers with distributed and geographically dispersed regional offices or storefronts tend to have a disjoint namespace environment.

In that environment, where the FQDN of a Windows server in AD does not match the FQDN of the same server in DNS, you will not be able to search the host’s DNS FQDN through VMM Add Host wizard, as it searches against your AD. More importantly, even if you selected “Skip AD query” option on the admin console, the regular “Add Host” job would still fail if required AD operation step is not taken. The root cause of the job failure is due to the fact that we need to use the DNS host name to connect to the host via WS-MAN, but the following authentication fails when we try to authenticate the host’s DNS FQDN against AD.

Hence, in order for hosts in this environment to be successfully added and managed by VMM, the instructions below should be followed:

· We recommend users to setup/install their VMM Server by making the VMM service running under a domain account (instead of the default server machine account);

· If you’re adding your host in disjoint namespace from Admin Console, at the 2^nd page of the AddHosts wizard, you will need to check the option “Skip Active Directory name verification”;

· The SPNs of the DNS host FQDNs must be added to AD;

· For clusters, you will need to add the hosts by using the “Add-VMHost” cmdlet (see sample code below).

PS C:\> $Credential = Get-Credential

PS C:\> Get-VMMServer -ComputerName "VMMServer.Contoso.com"

PS C:\> $VMHostGroup = Get-VMHostGroup | where {$_.Path -eq "All Hosts"}

PS C:\> Add-VMHostCluster -Name "VMHostCluster.Contoso.com" -VMHostGroup $VMHostGroup -RemoteConnectEnabled $TRUE -RemoteConnectPort 5900 -Credential $Credential

In VMM 2008 release, we require users to manually add the SPNs (use command “setspn –a” with appropriate AD privilege). Here is a reference Technet article with information on how to create SPNs. For a large environment, you may want to consult with your AD administrator to run a script to add the SPNs.

For VMM vNext, we’re working on a better solution, if the VMM server service account has AD privilege to add SPN, our AddHost logic will attempt to add the host’s DNS FQDN SPN when the authentication fails. With this new feature, if customers configure their AD properly, it will eliminate a lot of manual and tedious AD operations. Here is a KB article that talks about how to create AD permissions to allow a service account to add SPNs.

Hope this helps.

Thanks for reading!

Cheng

All about refreshers ...

Cheng Wei — Thu, 02 Oct 2008 02:19:00 GMT

If you have been using VMM for a while, you’ve probably figured out that we use various “refreshers” to pull data from the hosts and VMs that we manage. When deploying VMM in various sized environments, depending on the perf/scale requirements, you might need to tweak the refresher schedule to achieve the best performance.

So, to help you understand what’s happening “under the hood”, here are the refreshers we use in SCVMM 2008 product. Please note, that not all refreshers are customizable.

Host Refresher: Runs every 30mins on every host (with time lags) and updates host properties. It also updates disk/SAN and host network (NIC / virtual switch) information. It does not check any VM related properties or perf counters on hosts. Can be manually triggered on the UI or through cmdlet.

VM Refreshers: Two types

VM Heavy Refresher: Runs every 30mins on every Host and also runs whenever a VM is clicked on (or when you call refresh-vm cmdlet). It updates all VM properties, resource pools, clustering information for this VM and snapshots. But it does not update VM performance counter info.

VM Light Refresher: Runs every 2mins on every Host that is currently in the VMM database. This refresher does the following:

- Checks the host state (to see if the agent is running or the agent is down) and virtualization software status

- Syncs the state of a VM

- Marks VMs as missing

- Imports new VMs created outside of VMM (When VMM detects a new VM created on the virtualization platform, we import it into VMM and kick off the heavy refresher for the new VMs)

Cluster Refresher: Runs every 30mins and refreshes all cluster properties (including bringing in newly added nodes or taking our removed nodes). Can be manually triggered on the UI or through cmdlet.

Library Refresher: Runs on user configurable schedule (increment by 1 hour interval and can be turned off completely) and updates the library shares info and library objects. Can be manually triggered on the UI or through cmdlet.

Perf Refresher: Runs every 9 minutes on a host or whenever there is any state changing operation on the VM (i.e. start/stop/save/etc.). It collects perf counter information of both the host and all the VM’s on a given host.

VirtualCenter Refresher: Runs every 30mins and refreshes VC properties, ESX hosts and resource pools that are managed by this VC. Can be manually triggered on the UI or through cmdlet.

User Role Refresher: Runs every 30mins and updates user role properties.

PRO Tip Refresher: Runs every 30secs. It looks for PRO specific alerts in OpsMgr and reconciles the PRO tips in our DB against the data that is brought back from OpsMgr. This refresher cannot be manually triggered.

Thanks for reading and hope this is useful,

Cheng

Two important updates for SCVMM

Cheng Wei — Wed, 24 Sep 2008 20:52:00 GMT

Working with SCVMM team, Hyper-V team and BITS (Background Intelligent Transfer Service) team have just released two important updates to support SCVMM scenarios. We highly recommend our customers to install these updates to your Hyper-V host environments. In fact, if your Hyper-V hosts do not have these two updates installed, SCVMM server will report your hosts as "Needs Attention" (and the substatus for Virtualization Service Version will show as "Upgrade available").

Here are the brief discriptions (copied from the KB) and links to the updates:

Hyper-V Update:

Install this update to resolve potential issues when you manage Hyper-V with System Center Virtual Machine Manager (SCVMM). After you install this item, you may have to restart your computer. This update is provided to you and licensed under the Windows Server 2008 License Terms.

http://www.microsoft.com/downloads/details.aspx?FamilyID=FD44B4E3-2DCC-4299-B345-BC09A9A37B60&displaylang=en

BITS Update:

Install this update to resolve the scenario where a Background Intelligent Transfer Service (BITS) client cannot handle files that have paths that contain the volume GUID in Windows Server 2008. After you install this item, you may have to restart your computer. This update is provided to you and licensed under the Windows Server 2008 License Terms.

http://www.microsoft.com/downloads/details.aspx?FamilyId=9EC9DBB9-82AD-4D34-9267-76A0126A8F18&displaylang=en

Thanks for reading and hope this is helpful,

Cheng

Install and uninstall VMM agent on Windows 2008 Server Core

Cheng Wei — Mon, 08 Sep 2008 22:15:00 GMT

Since Windows 2008 Server Core does not come with Graphical UI, managing things on Core can be a bit challenging for folks who are used to the various administrator consoles.

Luckily, here is a handy dandy command reference: http://www.microsoft.com/downloads/details.aspx?familyid=5FB255FF-72DA-4B08-A504-1B10266CF72A&displaylang=en, in case you want to learn more about the available commands. Note, PowerShell is not available on Core since it needs .NET Framework.

So, how do you install and/or uninstall SCVMM agent on Core server? Well, it depends on what type of host this Server Core is:

· Install agent:

o If the Server Core system is to be added as a domain-joined server (not a perimeter network host), the easiest way to do so is to add it directly through VMM admin console. As long as you supply proper credentials for the VMM server to access the host, the agent should be installed for you by our AddHost logic.

o If the Server Core system is to be added as a perimeter network host, in which case a manual agent install is required, you would need to do two things:

§ Install the VC Redist first, which can be located on the VMM DVD at : [DVD]\prequisites\VCRedist\

§ Then install the VMM agent by launching the following MSI command:

· For 64-bit OS, msiexec.exe –I [DVD drive]\amd64\msi\Agent\vmmAgent.msi;

· For 32-bit OS, msiexec.exe –I [DVD drive]\i386\msi\Agent\vmmAgent.msi;

§ Once the installation starts, you will see a wizard that walks through the agent installation process, at which you can choose to install a perimeter network host agent.

· Uninstall agent:

o From Admin Console:

§ If the host is currently managed by VMM as a (trusted or untrusted) domain-joined host, the easiest way to uninstall the agent is to remove the host through VMM admin console. When the host is successfully removed from VMM server, the agent will be uninstalled from the host.

o Manual Uninstall from the host

§ If the host is managed as a perimeter network host, successfully removing the host from VMM does not uninstall the agent from the host. Users need to uninstall the agent from the host locally, just as how the agent was initially installed for perimeter network host.

§ For non-Core server, users can go to the “Programs and Features” and uninstall the agent from the UI.

§ For Core server, users will need to select one of the following two methods:

· If the original agent MSI file is still saved around, users can run “msiexec.exe /x <path>\vmmagent.msi” to uninstall the agent.

· If you no longer has the original MSI file for the agent installed on the host, run the command recorded in the registry:

1. Launch “regedit”

2. Go to node “HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\”

3. Search for “Microsoft System Center Virtual Machine Manager Agent”, and copy the value of the key “UninstallString”.

4. Run the command from an elevated command prompt.

Hope this is helpful and thanks for reading,

Cheng

How to collect WinRM traces

Cheng Wei — Fri, 29 Aug 2008 02:59:00 GMT

Since “How to collect SCVMM traces” and “What to do when I have hosts in Needs Attention state?” are published, we've seen requests on how to further debug WinRM/WMI issues that lead to VMM AddHosts failures.

It’s particularly useful, when you run into a SCVMM error that is rooted from a WinRM error code with a generic error message. So, to make your debugging effort easier, here is the process to collect WinRM traces, which hopefully provide more clues on the root cause of the WinRM failure on the remote host with help from your Microsoft support:

1) Download the Windows SDK toolkit:

1. For Windows 2008 servers:

1. If the server you’re trying to collect WinRM traces from has Internet access, download tracelog from: http://www.microsoft.com/downloads/details.aspx?FamilyId=E6E1C3DF-A74F-4207-8586-711EBE331CDC&displaylang=en;

2. if not, you can download the iso image (about 1.4G) of the Windows SDK from here and burn it into a DVD: http://www.microsoft.com/downloads/details.aspx?FamilyId=F26B1AA4-741A-433A-9BE5-FA919850BDBF&displaylang=en

2. For Windows 2003 or client (XP / Vista) systems:

1. Download it from if you have Internet access: http://www.microsoft.com/downloads/details.aspx?FamilyID=4377f86d-c913-4b5c-b87e-ef72e5b4e065&displaylang=en&Hash=voZ0UJrspn5qT6W4kcu%2fOMSjQf75oB2WCTbQ262yAcHxnRmEWyOw6qqQlesatQ8YZBNHBRZQxyYohHYvUNyNPA%3d%3d

2. Download the ISO file from: http://www.microsoft.com/downloads/details.aspx?FamilyId=F26B1AA4-741A-433A-9BE5-FA919850BDBF&displaylang=en

2) Install The Windows SDK to one of the folders that are in you default path.

3) Save the following text into a starttracing.cmd file to the folder where you install Windows SDK (in particular tracelog.exe).

@echo off

@echo on

REM Usage:

REM this.cmd - turns all tracing on

REM this.cmd 0xHHHH - turns tracing w/ for specific bits

REM see bits in WPP_DEFINE_CONTROL_GUID for more details

REM bits a counted from first entry (0th bit) to last entry (Nth bit)

REM

REM E.g.: this.cmd 0x1C00 will turn on tracing for automation component

REM

@echo off

set FLAGS=0x1

if NOT {%1}=={} (

set FLAGS=%1

)

@echo FLAGS=%FLAGS%

@echo ------------------------------------------------------------------------------------------------

@echo attempting to stop and re-start Wmxlog logger for logging

@echo ------------------------------------------------------------------------------------------------

tracelog.exe -stop Wmxlog

tracelog.exe -start Wmxlog -f wmxlog.etl -guid tracectl.txt -ft 1 -flag %FLAGS%

@echo ------------------------------------------------------------------------------------------------

@echo run your application now and then use displaytrace.cmd to see the traces logged

@echo ------------------------------------------------------------------------------------------------

goto End

:CtlNotExist

@echo tracectl.txt is missing

goto End

:NtTreeNotDefined

@echo variable _NTTREE is not defined

goto End

:TraceFormatDirNotExist

@echo directory %_NTTREE%\Symbols.pri\TraceFormat (_NTTREE\Symbols.pri\TraceFormat) does not exist

goto End

:End

4) Save the following text into a stoptracing.cmd file to the folder where you put tracelog.exe.

@echo off

@echo ------------------------------------------------------------------------------------------------

@echo stop Wmxlog logger

@echo ------------------------------------------------------------------------------------------------

tracelog.exe -stop Wmxlog

5) Save the following text into a tracectl.txt file to the folder where you put tracelog.exe.

04c6e16d-b99f-4a3a-9b3e-b8325bbc781e WinRM

c0a36be8-a515-4cfa-b2b6-2676366efff7 WinrsMgr

f1cab2c0-8beb-4fa2-90e1-8f17e0acdd5d WinrsExe

03992646-3dfe-4477-80e3-85936ace7abb WinrsCmd

6) Run “starttracing.cmd”.

7) Repro your steps that you see the WinRM failure.

8) Run “stoptracing.cmd”. The resulting wmxlog.etl is the trace for WinRM.

Please note that the trace is unreadable since it's binary data. It needs to be sent to your Microsoft support contact to render and explain what has happened in the recorded trace.

Hope this helps and thanks for reading,

Cheng

What ELSE can I do when my host is in Not Responding state?

Cheng Wei — Wed, 30 Jul 2008 21:23:00 GMT

After I posted "What to do when my host is in Not Responding state?", I received a few queries about the issues people ran into.

I captured my notes when explaining some of the errors folks reported. I thought that it might be interesting to share them, in case you ran into a similar issue and you would find this information useful.

====================================================

For the Add Hosts errors you encountered, let’s take them one at a time.

· For error 410, next time if you run into this error code, here are the steps I’d like you to try:

o It’s likely an WinRM issue during the agent install process.

o Immediately after the remote Add Host failure, try installing the agent locally on the host by using the following command: “msiexec.exe /i vmmAgent.msi /L*V c:\tmp\vmmagent_install.log”.

o If the agent install still fails, inspect the msi log file (c:\tmp\vmmagent_install.log) and see where the failure occurs or just send me the log file and we’ll help you to root cause the install failure.

Error (410)

Agent installation failed on IDC-HVH-02.xxxxxx.com.

(Fatal error during installation (0x80070643))

Recommended Action

Try the operation again. If the problem persists, install the agent locally and then add the managed computer.

· For error 415,

o This error is very straightforward – it simply means that when the VMM server tried to copy the agent install service executable over to the ADMIN$ share on the remote host (IDC-HVH-01.xxxxxx.com), the copy process failed.

o The troubleshooting steps are also straightforward:

§ See if you can manually access the ADMIN$ of the remote host form the VMM Server, if cannot, fix the network share issue.

§ See if the volume is full, if so extend the volume (or clean up some space).

§ See if there is a file there with the same name already, if so, remove / rename the file.

o If none of the steps help, we’ll need your VMM Server AND host traces to help you further troubleshoot.

Error (415)

Agent installation failed copying C:\Program Files\Microsoft System Center Virtual Machine Manager 2008\agents\I386\2.0.3194.0\msiInstaller.exe to \\IDC-HVH-01.xxxxxx.com\ADMIN$\msiInstaller.exe.

(The process cannot access the file because it is being used by another process)

Recommended Action

1. Ensure IDC-HVH-01.copaair.com is online and not blocked by a firewall.

2. Ensure that there is sufficient free space on the system volume.

3. Verify that the ADMIN$ share on IDC-HVH-01.xxxxxx.com exists. If the ADMIN$ share does not exist, reboot IDC-HVH-01.xxxxxx.com and then try the operation again.

· For error 415,

o This error indicates that the WinRM service on the host is not responding as what the VMM Server expects.

o Besides the basic diagnostic advices VMM team typically gives (check to make sure WinRM service is running, Listner is configured, firewall is not blocking the HTTP port, etc.), your best option is to contact WinRM team and get their expertise to exam the host to see why the WinRM service is not responding.

Error (407)

The agent operation with server IDC-HVH-01.xxxxxx.com was not successful.

(Unknown error (0x80338012))

Recommended Action

Try the operation again. If the problem persists, ensure that the VMM server is able to communicate with the managed computer.

FYI - C:\Users\chengwei>winrm helpmsg 0x80338012

The client cannot connect to the remote host specified in the request. Verify that

the service on the remote host is running and is accepting requests. You may

use the following command to analyze the state of the WinRM service and to configure

the service, if necessary: "winrm quickconfig".

====================================================

For the Host Refresh errors you encountered, let’s take look at them one at a time.

· For error 500 below, it seems to be a typical system resource issue with the host (or could be a WinRM internal issue).

o Typically for this error, you either need to restart WinRM service, or you might need to restart the machine, if the machine is really running out of system resources.

Error (500)

Could not retrieve configuration data from the virtual machine host IDC-HVH-02.xxxxxx.com.

(No more threads can be created in the system (0x800700A4))

Recommended Action

Verify that Windows Management Instrumentation, Windows Remote Management (WS-Management), Virtual Server, and Virtual Machine Manager Agent services are running on IDC-HVH-02.xxxxxx.com. Then try the operation again.

· For error 2915,

o It seems that WinRM cannot find the host.

o Try and see if the host name can be looked up in DNS. And the host SPN is registered in AD.

o If none of the steps help, we’ll need your VMM Server AND host traces to help you further troubleshoot.

Error (2915)

The WS-Management service cannot process the request. Object not found on the IDC-HVH-02.xxxxxx.com server.

(Unknown error (0x80338000))

Recommended Action

Ensure that the agent is installed and running. If the agent is installed, the WMI object may not have been found.

C:\Users\chengwei>winrm helpmsg 0x80338000

The WS-Management service cannot process the request. The service cannot find the

resource identified by the resource URI and selectors.

· For error 422,

o As what the error message says, either the machine is no longer in existence (name/IP changed, brought to other network or taken down for service?), or the machine is associated with other VMM server.

o Recommended next steps include:

§ Check to ensure that the machine is still available on the network (IP/name has not been changed).

§ Check to ensure that the computer is not associated with other VMM Server. If it’s accidentally associated with a different VMM Server, remove it from the other VMM Server and then re-add it back to this VMM server.

o If none of the steps help, we’ll need your VMM Server AND host traces to help you further troubleshoot.

Error (422)

Machine is not associated with this Virtual Machine Manager server.

Recommended Action

Check the machine name, and then try the operation again.

Hope this is helpful & thanks for reading!

Cheng

Removing an ESX host or stopping managing an ESX host?

Cheng Wei — Thu, 10 Jul 2008 00:01:00 GMT

We have heard some feedback that some of our customers did not expect VMM to remove an ESX host when a "Remove" action was requested for an ESX host managed through VirtualCenter Server (VC) by VMM Server. So, I think that it might be worth to talk about how exactly this works and how you could do what you intend to do on VMM:

How it works:

Users can add a new ESX (that is not managed by any VC) through VMM's "Add Host" wizard or Add-VMHost cmdlet, and when the host is added to VMM, it's also added to the associated VC.

Note: We do require users to add at least one VC before users can add any ESX host.

Users can delete an existing managed (by VMM) ESX host from VMM, and when the host is deleted from VMM, it's also deleted (really deleted) from the VC that the ESX host is associated with.

What you may want to do, if

if you want to delete the ESX host from VMware resource pool,

Delete it from VMM, so that the host will also be removed from VC and the resources on the host are taken out of the resource pool.

If you just want VMM to stop managing a specific ESX host that is added through VMM's "Add Host" wizard or auto-added when you added the VC,

Delete the VC that the ESX host is associated with from VMM.
Removing VC from VMM does not actually remove / destroy any VMware resources. It simply removes the information VMM Server knows about the VC and ESX hosts that are associated with this VC from VMM database.
Also note that when you remove the VC, all ESX hosts associated with this VC will also be removed from the VMM's view.

Hope this helps.

Thanks for reading,

Cheng

Managing Hosts in Windows 2000 AD Forest By Using VMM 2008

Cheng Wei — Thu, 05 Jun 2008 08:26:00 GMT

As you know when you install VMM 2008 server, the computer where you want to install VMM 2008 Server must be joined in a Windows 2003 or Windows 2008 AD domain. That's a hard requirement and the install will be blocked if it is not met.

We hope all our customers have migrated their AD infrastructure from Windows 2000 AD to Windows 2008 or, at least, Windows 2003 version. There is just a little over two years before Windows 2000 server is complete out of support (7/13/2010).

However, in case you are one of the few customers who have not fully migrated, or are still in process of migrating your Windows 2000 AD infrastructure to a recent version, here is what you could try to temporarily workaround this limitation before or during your AD migration:

If you have not done forest-prep, perform forest-prep in the Windows 2000 AD forest and get the schema updated to Windows 2003 or 2008 AD schema.

Note: There will be result in short-term replication traffic overhead in your network, since Windows 2000 AD cannot replicate deltas of the PAS.

Introduce a separate Windows 2003 or Windows 2008 AD site in your Windows 2000 AD forest that will contain the VMM Server and Hyper-V host servers.
Configure two-way trust between the Windows 2003 or Windows 2008 AD domain controller and the Windows 2000 AD forest domain controllers.
Through AD site topology and DNS enforcement, ensure that VMM Server only authenticate with the Windows 2003 or the Windows 2008 domain controller.

After above configuration, you should be able to add hosts that live in a Windows 2000 AD forest into the VMM server that is in a Windows 2003 or Windows 2008 AD site.

Here is a reference document on how to set up a separate AD site for Exchange:Microsoft IT Showcase: Creating an Active Directory Site for Exchange Server

Thanks for reading!

Cheng

Unable to add host or hosts in "Needs Attention" state (SCVMM 2008 Beta)?

Cheng Wei — Fri, 02 May 2008 20:53:00 GMT

These are probably two of the most frequently asked questions in the Technet SCVMM forum.

I'm unable to add host, what did I do wrong?
What do I need to do when I have host in "Needs Attention" state or my "Connection Status" is "Not Responding"?

The first thing I'd ask people to do is to check the FAQ published on Connect site along with the beta bits, which covers the known issues and limitation for this beta release. One of the things we documented there is that a USB drive could cause your host be in this "Needs Attention" state. We're fixing the issue for our RTM release.

Secondly, I'd recommend you to check the troubleshooting note published for our VMM 2007 release, which covers a lot of commonly seen and known issues for both add host (or agent install) scenarios and hosts in "Needs Attention" state. The most commonly seen causes of failure are:

Host Hyper-V service is not working properly. The causes include:

Host computer's hardware is not virtualization capable (check here for what Hyper-V requires for hardware-assisted virtualization).
Host computer's hardware virtualization is not enabled. What we found is that most Intel chip based computer's VT is not enabled in factory default, AMD-chip-based computer's is enabled by default.
In either case, you might need to check the BIOS to confirm.

Firewall rules on the host or in the network blocks the initial DCOM communication, or HTTP/HTTPS communications between VMM server and the to-be-managed host.

Turning off the firewall during the initial deployment may help identify the causes of AddHosts failure.
Please note, once the agent is deployed, SCVMM server does not use DCOM to communicate with the hosts. Instead, we use HTTP(for WS-MAN)/HTTPS(for BITS) to communicate with the hosts. As part of the agent install, we add firewall exceptions to allow HTTP and HTTPS traffic to the host. Therefore, once the host agent is deployed, as long as your HTTP and HTTPS ports are open, VMM server should be able to connect to your hosts.

SCVMM user and group was removed for DMZ host or untrusted domain joined host.

During agent deployment, SCVMM creates a group SCVMM and a user starting with string "SCVMM". And the user "SCVMMxxxxx" is added to the local Administrators group and the Virtual Machine Manager Servers group.
We've seen cases where the user and group created by SCVMM are removed by group policy automatically, or removed manually by mistake. Without this user, VMM server will not be able to connect to the host.
Please note, in trusted domain joined host scenario, we do not create local user. We use VMM Server machine account to authenticate and communicate with the host.

Incorrect version of Hyper-V

For this SCVMM 2008 Beta release, we support Hyper-V RC0 release (KB949219).

Lastly, here are some additional testing you could do on your host to unveil issues with services that VMM is dependent of:

To check integrity of or repair your host WMI repository,

run "winmgmt /salvagerepository".

To check your WMI virtualization store,

start wbemtest by using a local admin account of the host machine and
connect to \\<host>\root\virtualization
Try query "select * from msvm_ComputerSystem" and see if it succeeds.

To check your host's WMI namespace security settings,

Go to Server Manager, click on Configuration then right-click on WMI Control and choose Properties
Click on Security tab, choose SCVMM and check its Security Settings.

Make sure that both Administrators group and Virtual Machine Manager Servers group have the full rights on the namespace.

To check your WnRM state,

run "winrm qc".

To check if the WinRM listener is running,

run "winrm enum winrm/config/listener", or
run "winrm get winrm/config" to get the entire WinRM config.

To check if your VMM agent is responding to winrm calls,

run "winrm invoke GetVersion wmi/root/scvmm/AgentManagement -r:http://<host> @{}".

To manually restart VMM server service:

run "net stop vmmservice", and run "net start vmmservice".

To manually restart VMM host agent service:

run "net stop vmmagent", and run "net start vmmagent".

To manually restart Hyper-V service:

run "net stop vmms", and run "net start vmms".

Debugging

If above resources cannot help you find the root cause, try collect debugging traces by following the instructions posted here, study the trace logs and look for exceptions from the traces to get a clue on where things might have gone wrong.

Hope this helps and thanks for reading,

Cheng

Add host into VMM 2008 Beta

Cheng Wei — Thu, 01 May 2008 02:56:00 GMT

Well, once you finish installing VMM Server and have your admin console setup, the next you want to do is to start managing your hosts. Virtual Server hosts, Hyper-V hosts and VMware ESX hosts.

I talked about managing VMware VCS and how VMM imports the existing ESX hosts managed by the VCS. Next, let me drill into how you can start adding Virtual Server hosts and Hyper-V hosts, as well as some AddHost limitations in this Beta release.

This is what you will see in the Add Host wizard from the admin console for our VMM 2008 beta release:

(Note: This UI will change by our RTM.)

First, here are some requirements for adding hosts with different types of virtualization platforms:

To add a Virtual Server host

You must have Windows Server 2003.
In beta release, we do not support Virtual Server running on Windows Server 2008.
We're looking to provide support for running Virtual Server on Windows Server 2008 in our RTM release.
The Virtual Server needs to be at version 2005 R2 SP1 release.

To add a Hyper-V host

You must have Windows Server 2008.
Your computer must meet the hardware virtualization assistance requirements:

64-bit CPU
VT (Virtualization Technology feature from Intel) or AMD-V (from AMD) enabled on the BIOS
NX (No eXecute) /XD (eXecute Disable)
And full BIOS support for hardware virtualization

You must install Hyper-V RC0 software update.

To add a VMware ESX host

You must have at least one VMware VirtualCenter Server added to VMM.
Your VirtualCenter Server must be running version 2.0 or above.
Your ESX host must be running version 3.0 or 3.1.

Here are the things your need to know for different types of server (server locations):

To add a trusted domain joined Windows host

Make sure the requirements for the virtualization platform are met (see above).
Make sure the server is domain joined.
Make sure your server is reachable (DNS, AD, network firewall, etc.).
Make sure you know the proper credentials you need to let VMM use to manage the host.
Note: When there is no two-way trust between the VMM server domain and the domain the Windows host (to be added), you will need to use to do it from the CLI (Add-VMHost cmdlet) by using the "-NonTrustedDomainHost" switch. In our Beta release, you cannot do this from the UI.

To add a perimeter network (or DMZ) Windows host or a non-domain-joined Windows host

Make sure the requirements for the virtualization platform are met (see above).
Make sure the agent is installed locally, and security file is generated and transfered to the VMM console.
Make sure your server is reachable (DNS, network firewall, etc.).

To add a VMware ESX host

Make sure you have at least one VMware VirtualCenter Server added to VMM.
Make sure you know the "root" user credential.

Thanks for reading,

Cheng