Cheng's Random Thoughts on System Management : VMware

Got a "Bad Gateway"?

Cheng Wei — Tue, 14 Apr 2009 19:41:00 GMT

Greetings, folks!

Hope you had a great Easter on last Sunday, if you celebrate that holiday! My kids certainly enjoyed hunting eggs in a typical Seattle drizzling Saturday morning. J

This blog entry is for folks who use or plan to use VMM to manage VMware ESX hosts.

If this is your first time adding your ESX host to VMM, and your ESX hosts are added into VMM by using DNS name (instead of IP address), and your network environment with Internet proxy is not made aware of VMware host DNS names, you may see error similar to below:

Error (2891)

VMM cannot complete the …. action on the vmm-esx1.scvmm.contoso.com server because of the error: The remote server returned an error: (502) Bad Gateway.

(Unknown error (0x194))

This issue may occur when:

· To turn your ESX host from “OK(Limited)” state to “OK” state (to get your ESX host in full management mode), you are providing root user credentials and attempting to retrieve the certificate and SSH key from the ESX host to approve.

· You may also run into the same error when attempting to perform a live migration (VMotion) for a VM hosted by an ESX host from VMM admin console.

The cause of this issue is due to the DNS resolution (or proxy configuration) on your VMM server:

· VMM uses web services to communicate to VMware Virtual Center and ESX hosts.

· Proxies are generally configured to permit pass-through of local DNS names.

· In some cases, when the proxy is not configured for a DNS name, the web service connection may fail.

Following are some simple options to resolve this issue:

· SOLUTION 1: Include VMWare host DNS names in the internet proxy configuration.

· SOLUTION 2: Add dns hosts to the proxy configuration of Internet Explorer

· SOLUTION 3: Create a file "vmmservice.exe.config" in the same location as vmmservice.exe (by default, it’s installed under this directory: %ProgramFiles%\Program Files\Microsoft System Center Virtual Machine Manager 2008\bin\) with contents similar to the following. Note: you will need to bounce the VMM server service (restart “Virtual Machine Manager” service from services.mcs) for it to pick up the new proxy configuration).

<system.net>

</bypasslist>

</defaultProxy>

</system.net>

</configuration>

On the extreme side, you could also completely turn off the proxy on your VMM server by applying the following config file:

<system.net>

</defaultProxy>

</system.net>

</configuration>

Hope this helps and thanks for reading!

Cheers,

Cheng

All about refreshers ...

Cheng Wei — Thu, 02 Oct 2008 02:19:00 GMT

If you have been using VMM for a while, you’ve probably figured out that we use various “refreshers” to pull data from the hosts and VMs that we manage. When deploying VMM in various sized environments, depending on the perf/scale requirements, you might need to tweak the refresher schedule to achieve the best performance.

So, to help you understand what’s happening “under the hood”, here are the refreshers we use in SCVMM 2008 product. Please note, that not all refreshers are customizable.

Host Refresher: Runs every 30mins on every host (with time lags) and updates host properties. It also updates disk/SAN and host network (NIC / virtual switch) information. It does not check any VM related properties or perf counters on hosts. Can be manually triggered on the UI or through cmdlet.

VM Refreshers: Two types

VM Heavy Refresher: Runs every 30mins on every Host and also runs whenever a VM is clicked on (or when you call refresh-vm cmdlet). It updates all VM properties, resource pools, clustering information for this VM and snapshots. But it does not update VM performance counter info.

VM Light Refresher: Runs every 2mins on every Host that is currently in the VMM database. This refresher does the following:

- Checks the host state (to see if the agent is running or the agent is down) and virtualization software status

- Syncs the state of a VM

- Marks VMs as missing

- Imports new VMs created outside of VMM (When VMM detects a new VM created on the virtualization platform, we import it into VMM and kick off the heavy refresher for the new VMs)

Cluster Refresher: Runs every 30mins and refreshes all cluster properties (including bringing in newly added nodes or taking our removed nodes). Can be manually triggered on the UI or through cmdlet.

Library Refresher: Runs on user configurable schedule (increment by 1 hour interval and can be turned off completely) and updates the library shares info and library objects. Can be manually triggered on the UI or through cmdlet.

Perf Refresher: Runs every 9 minutes on a host or whenever there is any state changing operation on the VM (i.e. start/stop/save/etc.). It collects perf counter information of both the host and all the VM’s on a given host.

VirtualCenter Refresher: Runs every 30mins and refreshes VC properties, ESX hosts and resource pools that are managed by this VC. Can be manually triggered on the UI or through cmdlet.

User Role Refresher: Runs every 30mins and updates user role properties.

PRO Tip Refresher: Runs every 30secs. It looks for PRO specific alerts in OpsMgr and reconciles the PRO tips in our DB against the data that is brought back from OpsMgr. This refresher cannot be manually triggered.

Thanks for reading and hope this is useful,

Cheng

Removing an ESX host or stopping managing an ESX host?

Cheng Wei — Thu, 10 Jul 2008 00:01:00 GMT

We have heard some feedback that some of our customers did not expect VMM to remove an ESX host when a "Remove" action was requested for an ESX host managed through VirtualCenter Server (VC) by VMM Server. So, I think that it might be worth to talk about how exactly this works and how you could do what you intend to do on VMM:

How it works:

Users can add a new ESX (that is not managed by any VC) through VMM's "Add Host" wizard or Add-VMHost cmdlet, and when the host is added to VMM, it's also added to the associated VC.

Note: We do require users to add at least one VC before users can add any ESX host.

Users can delete an existing managed (by VMM) ESX host from VMM, and when the host is deleted from VMM, it's also deleted (really deleted) from the VC that the ESX host is associated with.

What you may want to do, if

if you want to delete the ESX host from VMware resource pool,

Delete it from VMM, so that the host will also be removed from VC and the resources on the host are taken out of the resource pool.

If you just want VMM to stop managing a specific ESX host that is added through VMM's "Add Host" wizard or auto-added when you added the VC,

Delete the VC that the ESX host is associated with from VMM.
Removing VC from VMM does not actually remove / destroy any VMware resources. It simply removes the information VMM Server knows about the VC and ESX hosts that are associated with this VC from VMM database.
Also note that when you remove the VC, all ESX hosts associated with this VC will also be removed from the VMM's view.

Hope this helps.

Thanks for reading,

Cheng

Add host into VMM 2008 Beta

Cheng Wei — Thu, 01 May 2008 02:56:00 GMT

Well, once you finish installing VMM Server and have your admin console setup, the next you want to do is to start managing your hosts. Virtual Server hosts, Hyper-V hosts and VMware ESX hosts.

I talked about managing VMware VCS and how VMM imports the existing ESX hosts managed by the VCS. Next, let me drill into how you can start adding Virtual Server hosts and Hyper-V hosts, as well as some AddHost limitations in this Beta release.

This is what you will see in the Add Host wizard from the admin console for our VMM 2008 beta release:

(Note: This UI will change by our RTM.)

First, here are some requirements for adding hosts with different types of virtualization platforms:

To add a Virtual Server host

You must have Windows Server 2003.
In beta release, we do not support Virtual Server running on Windows Server 2008.
We're looking to provide support for running Virtual Server on Windows Server 2008 in our RTM release.
The Virtual Server needs to be at version 2005 R2 SP1 release.

To add a Hyper-V host

You must have Windows Server 2008.
Your computer must meet the hardware virtualization assistance requirements:

64-bit CPU
VT (Virtualization Technology feature from Intel) or AMD-V (from AMD) enabled on the BIOS
NX (No eXecute) /XD (eXecute Disable)
And full BIOS support for hardware virtualization

You must install Hyper-V RC0 software update.

To add a VMware ESX host

You must have at least one VMware VirtualCenter Server added to VMM.
Your VirtualCenter Server must be running version 2.0 or above.
Your ESX host must be running version 3.0 or 3.1.

Here are the things your need to know for different types of server (server locations):

To add a trusted domain joined Windows host

Make sure the requirements for the virtualization platform are met (see above).
Make sure the server is domain joined.
Make sure your server is reachable (DNS, AD, network firewall, etc.).
Make sure you know the proper credentials you need to let VMM use to manage the host.
Note: When there is no two-way trust between the VMM server domain and the domain the Windows host (to be added), you will need to use to do it from the CLI (Add-VMHost cmdlet) by using the "-NonTrustedDomainHost" switch. In our Beta release, you cannot do this from the UI.

To add a perimeter network (or DMZ) Windows host or a non-domain-joined Windows host

Make sure the requirements for the virtualization platform are met (see above).
Make sure the agent is installed locally, and security file is generated and transfered to the VMM console.
Make sure your server is reachable (DNS, network firewall, etc.).

To add a VMware ESX host

Make sure you have at least one VMware VirtualCenter Server added to VMM.
Make sure you know the "root" user credential.

Thanks for reading,

Cheng

VMM 2008 Architecture

Cheng Wei — Thu, 01 May 2008 02:21:00 GMT

If you are in MMS 2008, you've probably seen this architecture diagram:

For folks who did not get a chance to go, here are some of the key architectual characteristics that I'd like to point out:

On the top-left corner, notice that the UIs (admin console and self-service portal) are all 100% built on top of Powershell. What does this mean? It means that

Everything you do on the UI can be done through powershell. All actions/jobs are 100% scriptable.
In addition, if you are a "script-junkie", :-), you can do more via our powershell interface than what's made available on the UI. Yes, power to Powershell!

Notice the "Connector" sitting between VMM and SCOM, which is part of the new PRO (Performance and Resource Optimization) feature, and that allows a faster message/information sharing between VMM and SCOM.
In the mid of the picture, notice the horizontal single layer of "Management Interfaces". What is so interesting about that?

We have abstracted different virtualization platforms and their APIs, and present a set of consistent and platform-agnostic cmdlets and UI interfaces to perform your heterogeneous host and VM management operations.
It's totally transparent (I mean "no op" here) to you, no matter you are sending an instruction to a Windows Hyper-V host, a VMware ESX host or a Virtual Server host. This is what we call simplified management experience. :-)

On the lower-right corner, it shows how we're managing VMware ESX hosts via VCS (VirtualCenter Server). Most of the host and VM operations are done directly with VCS from VMM server. Note that VMM server may still need to communicate with ESX hosts directly when there is need for file-level enumeration (for example, Save State or Clone VM, etc.).

Thanks for reading,

Cheng

Managing VMware

Cheng Wei — Wed, 30 Apr 2008 07:12:00 GMT

Why should I use VMM to manage my VMware?

In Rakesh's post, he talked extensively on why we decided to manage VMware. From customer value prop point of view, I just wanted to highlight one of the key values SCVMM 2008 brings -- the ability to provide a "single-pane-of-glass" view into your virtualized environment.

A single console to go to, a single view to manage your heterogeneous virtualized assets, a single tool to learn and use.

How does it work and where do I start?

The workflow of managing VMware starts from adding your VMware VirtualCenter Server (VCS) into VMM. "Add a VirtualCenter Server" is a global action that is available on all views of the admin console. When initiated, you will be asked to provide the administrator's credential for the VirtualCenter Server. Once the VCS is managed by VMM (BTW, the management is achieved without installing agent on VCS - we're using VMware published webservices APIs to manage VCS and ESX hosts), VMM automatically pulls all the ESX host records from the VCS server and puts them into VMM's host view.

At this point, a new host state is created for the newly imported ESX hosts - "OK (Limited)". Why is it not in OK state? When we import all the ESX host records from VCS, we don't have access to the host credentials encrypted and stored in VCS data store. Hence, it's recommended that, after the hosts are imported into VMM, you go to the Hosts view and provide credentials for the hosts that are in "OK (Limited)" state. Once the credentials are provided and verified, the host state will turn to "OK" state, where the full set of VM operations are supported. Please note, a subset of VM operations that do not require file-level operations can still be performed for an "OK (Limited)" ESX host. The table in the next section lists out the exact operations that are supported in each modes.

Here is a sample view of what the host with credentials and one without may look like in VMM console.

Here are the UI controls for providing credentials. Once proper credentials are provided, the host will go into OK state.

When you need to add additional ESX hosts into the VMM environment afterwards, you can do so by using the AddHost wizard, which I will cover in a different post later. One thing to note is that a VCS must be added to VMM before a new ESX host can be added.

What are the operations that can be done on an "OK"-state ESX host versus an "OK (Limited)" ESX host?

The table below shows the comparison of VM operations allowed for each states (this table is also available in the FAQ posted on the Connect site):

Virtual machine action	No host credentials and host SSH root disabled	Host credentials provided and host SSH root enabled
Virtual machine action	OK (Limited) Status	OK status
Start	Y	Y
Stop	Y	Y
Pause	Y	Y
Save State	N	Y
Discard saved state	N	Y
Properties	Y	Y
New checkpoint	Y	Y
Manage checkpoints	Y	Y
Remove virtual machine	Y	Y
Migrate(Vmotion)	Y	Y
Migrate VM (Across virtual center)	N	Y
Store in Library (migrate a virtual machine from an ESX Server host to the VMM library)	N	Y
Clone (within same virtual center)	N	Y
Clone virtual machine (on same ESX Server host)	N	Y
Convert virtual machine (V2V)	N	Y
New virutal machine(from VMM template)	N	Y
New virtual machine (Create new blank virtual hard disk.)	N	Y

Thanks for reading,

Cheng

System Center Virtual Machine Manager 2008 Beta has been shipped!

Cheng Wei — Tue, 29 Apr 2008 20:29:00 GMT

Greetings!

Earlier this morning, at MMS 2008 (Las Vegas), we announced that SCVMM 2008 (vNext) Beta is released and available for download at Connect site immediately. Here is the official press release. More news and pictures about our beta release can be found here.

VMM vNext Beta is a much awaited release, which provides supports for three virtualization platforms (Virtuall Server, Hyper-V and VMware) and offers some exciting new features. Here is a quick overview of the features we offer in this beta release:

Continue extending our rich feature sets from SCVMM 2007:

Host Configuration
Library Management
Virtual Machine Creation
Conversions: P2V & V2V
Delegation and Self Service
Intelligent Placement
Deployment and Storage Management
Monitoring and Reporting
Automation with PowerShell

New investment in SCVMM 2008 (vNext):

Heterogeneous Hypervisor Management (Hyper-V and VMware)
Windows Server 2008 Failover Cluster Integration
Performance & Resource Optimization (with integration with SCOM)
Delegated Administration

I'll drill into some of the new features in the future posts.

Thanks,

Cheng