HPC, Virtualization and Random Thoughts : Hyper-V

Live Migration, Cluster Shared Volumes & Networks

gmarchetti — Fri, 02 Oct 2009 02:42:49 GMT

The recommendation for people setting up live migration clusters is to isolate different kinds of traffic on their own networks:

Public network to access the cluster and the virtual machines running on it
“Private” cluster heartbeat network
“live migration” network
iSCSI network, if required to access shared storage

How do we determine what traffic goes where?

For public and private, the failover cluster manager tool is self-explanatory:

We select the appropriate cluster network properties. If we want to limit such network to private traffic, we do not allow clients to connect through it.

If we don’t want the cluster to use such network at all, e.g. because it is dedicated to iSCSI, we select the “Do not allow…” button.

How about the live migration traffic, though? It can be quite heavy, as we are copying memory pages from one host to another. We can select in which order to use cluster networks for such traffic through the failover cluster manager

The property requires some digging: expand “services and applications”, select the virtual machine in question, then in the main panel right-click on “virtual machine <name>” and you’ll see tab called “network for live migration”. You can then select and sort in order of priority the networks that you want to use. By default, live migration will select a network that is NOT used for CSV traffic. Note that you may have networks in this panel that were not selected for cluster use before. If you use iSCSI, de-select the relevant entry to make sure that the live migration traffic does not go through that network.

This brings me to cluster shared volumes. One of the great features of CSVs is that if the storage link (iSCSI, fibre) becomes unavailable for any reason on a node, storage traffic can be redirected over the cluster network to another node and hence to the storage device. But which cluster network?

Inter-node communications and CSV traffic will use the available network authorized for cluster use that has the lowest metric value. We can see the metrics with old cluster.exe

C:\Windows\system32>cluster net /prop
Listing properties for all networks:

T Network              Name                     Value
-- -------------------- ------------------------ -----------------------
SR Cluster Network 1    Name                     Cluster Network 1
MR Cluster Network 1    IPv6Addresses            <cut on purpose>
MR Cluster Network 1    IPv6PrefixLengths        <..>
MR Cluster Network 1    IPv4Addresses            <cut on purpose>
MR Cluster Network 1    IPv4PrefixLengths        <..>
SR Cluster Network 1    Address                  <..>
SR Cluster Network 1    AddressMask              <..>
S Cluster Network 1    Description
D Cluster Network 1    Role                     3 (0x3)
D Cluster Network 1    Metric                   10001 (0x2711)
D Cluster Network 1    AutoMetric               0 (0x0)
SR Cluster Network 2    Name                     Cluster Network 2
MR Cluster Network 2    IPv6Addresses
MR Cluster Network 2    IPv6PrefixLengths
MR Cluster Network 2    IPv4Addresses            <..>
MR Cluster Network 2    IPv4PrefixLengths        <..>
SR Cluster Network 2    Address                  <..>
SR Cluster Network 2    AddressMask              <..>
S Cluster Network 2    Description
D Cluster Network 2    Role                     1 (0x1)
D Cluster Network 2    Metric                   1000 (0x3e8)
D Cluster Network 2    AutoMetric               1 (0x1)

Note the 3 values:

Role: 1 for a private network, 0 for ignored by cluster, 3 for mixed traffic
Metric: the “weight” of the connection, generally in the 10,000 range for public networks, 1,000 for private ones. If a network has a default gateway, it is considered public; if not, private. Should there be more than one private or public network, the metric is incremented by 100 in order of enumeration (e.g. private network 2 will have a default metric of 1,100)
Autometric: 1 if the metric is set automatically by the cluster, 0 if you have set it manually.

So in my simple case the heartbeat network will also be used for CSV traffic. If you have more than 1 private network and you want to prioritize them, you can set the metric with cluster.exe, e.g.

C:\Windows\system32>cluster net "Cluster Network 2" /prop metric=1001

C:\Windows\system32>cluster net "Cluster Network 2" /prop

Listing properties for 'Cluster Network 2':

T Network              Name                     Value
-- -------------------- ------------------------ -----------------
SR Cluster Network 2    Name                     Cluster Network 2
MR Cluster Network 2    IPv6Addresses
MR Cluster Network 2    IPv6PrefixLengths
MR Cluster Network 2    IPv4Addresses                 <..>
MR Cluster Network 2    IPv4PrefixLengths             <..>
SR Cluster Network 2    Address                       <..>
SR Cluster Network 2    AddressMask                   <..>
S Cluster Network 2    Description
D Cluster Network 2    Role                     1 (0x1)
D Cluster Network 2    Metric                   1001 (0x3e9)
D Cluster Network 2    AutoMetric               0 (0x0)

Redirection of the traffic is automatic: if a network becomes unavailable, the next-lowest-metric one will be used. If another network with a lower metric becomes available, it will be used from that point onwards.

In Summary

By default, live migration traffic will be put on the network with the second-lowest metric. CSV traffic will be put on the the network with the lowest metric. In this simple example, I just have a public and private network, so the public one is used for live migration and the private one for csv and cluster traffic.

P2V with SCVMM – a quick tip

gmarchetti — Wed, 22 Jul 2009 00:54:53 GMT

System Center Virtual Machine Manager (SCVMM) has been offering a relatively simple way of doing physical-to-virtual migrations (P2V) for a while. You just click on the “Convert Physical Server” icon and off you go. Despite the name, it also works with client target machines. It’s simple, if you do some preparation work before.

In fact, VMM will ask you for name or ip address of the machine in question and for administrator credentials on it. Those will be used to reach the machine and install a p2v agent on it. For the process to work correctly, you must let through the firewall of the target machine:

WMI traffic
http
file and print
remote management

Also, make sure that the ADMIN$ share exists and start the Windows remote management service on the target machine.

By default, most of these ports and services are closed.

A Free Book on Microsoft Virtualization

gmarchetti — Fri, 13 Mar 2009 02:01:59 GMT

Understanding Microsoft Virtualization Solutions from Microsoft Press is available as a FREE download.

This 15MB E-Book gives an overview of all current Microsoft Virtualization technologies: Hyper-V, the Microsoft Enterprise Desktop Virtualization (MED-V), and VDI. It also describes which management solutions are available for them (e.g. System Center Virtual Machine Manager) and how they fit together. It is worth reading when planning the virtualization of your infrastructure.

You can find it here: http://csna01.libredigital.com/?urmvs17u33

Live Migration in R2

gmarchetti — Thu, 12 Feb 2009 21:00:00 GMT

I've got a lot of questions about Live Migration in 2008 R2. Rather than writing a long post on it, I thought I'd point you at some resources I found useful whilst setting up my test environment, so you can build one too:

- Frank Cicalese's (virtualization specialist) paper on r2 lab setup

- Technet step-by-step guide to live migration

- Technet guide to hyper-v fail-over

- Massimo Re Ferre` (IBM) tutorial on Hyper-V Server R2

- 2008 R2 Live Migration Architecture Guide

I also recorded the steps to build such environment in a series of screencasts that will be appearing on http://edge.technet.com. The first one is already there, so check it out. They are short out of necessity, so it will take a couple of weeks for all of them to appear.

What is new in virtualization with Windows Server 2008 R2?

gmarchetti — Thu, 13 Nov 2008 01:43:00 GMT

There are some quite interesting improvements in Windows Server 2008 R2 (what was wrong with W7 as a name?) that help us progress toward a dynamic infrastructure. Three of them are worthy of highlighting: live migration of virtual machines in hyper-v, cluster shared volumes and core parking.

1. Live Migration

Live migration refers to the ability of moving a running virtual machine from one host server to another without loss of service. For this to happen, we have to transfer the current virtual machine state and memory pages between machines and we have to warrant both servers the same level of access to the virtual machine files. The process can be summarized as follows:

Create a virtual machine on the target server
Copy the memory pages of the running virtual machine in question from the source to the target server via Ethernet. While we copy, those memory pages may change, so after an initial pass we have to go back and copy the changed set again, until a minimum threshold number of pages is reached. It is hard to fix the threshold: ideally, it will be the number of pages that can be copied within a TCP connection timeout, so the clients won’t notice.
Pause the source machine; copy its state across.
Resume the target machine, issue ARP command to update routing tables.

For (3) to happen quickly and transparently to the clients, the target server must have immediate access to the virtual machine files. It cannot wait for a disk volume to fail-over and possibly go through file system checks. That’s where cluster shared volumes come in.

2. Cluster Shared Volumes

Cluster Shared Volumes enable concurrent access to the same LUN by several nodes. Consequently, all the nodes see the same NTFS file-system and namespace. By the way, CSV is not a parallel or a cluster file system. It was designed with the live migration scenario in mind.

Since the host servers already mount the CSV, there is no need to arbitrate for disk access and fail over the volume hosting the virtual machine files. All you need to do is transfer ownership of those files and their locks to the target server.

CSVs are implemented via a filter driver mechanism, which is used to establish the access path to the underlying LUNs. This also enhances our fail-over ability, as file system requests will be redirected over the network to another server if a direct SAN access is no longer available.

3. Core idling or parking

Changes in Windows 7 power management allow for “density” scheduling, i.e. minimizing the number of processor cores on which work is done, hence maximizing their utilization. The idle cores can be put to sleep (low-power state Cx under the ACPI specifications), thus reducing power consumption. Hyper-V can take advantage of this feature and schedule its virtual machines accordingly. Power management policies can be controlled via WMI, policies and scripts.

If you combine “density” scheduling with the ability to move virtual machines among hosts, you achieve quite a scalable, efficient and dynamic solution to the distributed resource allocation problem. Now, all that remains to do is automate it. Stay tuned.

4. References

ACPI explanation on Wikipedia

WinHEC 2008 conference whitepapers

Engineering Windows 7 blog

The Windows blog

The Windows Server 2008 R2 Reviewers' Guide http://www.microsoft.com/windowsserver2008/en/us/r2.aspx

Stop Climate Change?! – Part 2

gmarchetti — Tue, 22 Jul 2008 03:41:00 GMT

I have been investigating some more in the area of Green IT, S+S. Some ideas and a lot of questions have come to mind. Please read on and let me know if they make any sense.

By the way, Part 1 is here :-)

1. How do you understand the status quo?

This may prove to be the most difficult part of the job. There aren’t many tools available. System Center Operations Manager, plus a few OEM management packs, are a starting point. Alas, you must build your own model to establish correlations between power utilization measured and applications over time. From those, you can derive a measure of efficiency. A few 3^rd-party applications (e.g. Verdiem’s Surveyor, Avocent and APC InfrastruXure) do a better job of establishing the baseline, although again they do little for the correlation analysis.

IBM’s Active Energy Manager goes a step further (on IBM hardware) by allowing you study trends and to take action on specific energy-related conditions. Again, it is not a complete “IT intelligence” tool.

2. How do you design your infrastructure and applications to optimize consumption?

Once you understand what type of load consumes what power (no small feat):

1. Can you reduce the physical tiers of your architecture? For instance, if you have a memory-intensive application and a CPU-intensive one, you may want to co-host them, thus using all the available cores and saving a few machines’ worth of power. This will only work from a performance point of view if you manage resource allocation tightly to avoid contention. In our example, you would run a thread belonging to the memory intensive application on 1 core and a thread of the cpu-intensive one on the other core of the same CPU socket. Before embarking on such a consolidation exercise, you will want to estimate the costs and the savings, in terms of power and money. Also keep in mind that as a consequence of the changed workload, you may require different hardware (e.g. “whole machines” rather than just blades) to optimize your power consumption profile over time.

2. Can you reduce the logical tiers of your architecture? Here’s an example: your application may use Sharepoint as a front-end, windows workflow to manage business logic, SQL for data processing, all running on separate hardware. Sharepoint can host workflows. SQL handles workflows in Integration Services and it can host an in-process CLR. With some clever re-architecting of your application, you may be able to get rid of the middle tier by using some combination of the two workflow services. The whole area of “power-conscious” applications is yet to be explored. We’re investigating.

3. Can you offload a tier of your architecture? Here’s where Software + Services comes into play. For instance, you may consider using an on-line storage service (e.g. SQL Server Data Services, aka CloudDB or Sitka) instead of hosting your own SQL. If you have a compute-intensive application, you may want to farm it out to a HPC provider and pay by CPU cycles utilized (Microsoft will offer such a service, now in pilot stage with a few ISVs). If your provider is able to consolidate several users’ workloads on its servers and charge for capacity consumed, the overall carbon footprint may be reduced – along with your costs.

4. If you do offload a function, how do you measure its performance against SLAs? This is actually the most difficult point. Technology is available to do all of the above (although not necessarily on Windows). Capacity-on-demand, for instance, has been a feature of certain Mainframes and Unix systems for years. Hosted services offering are widely available. However, different security boundaries and political pressures make it difficult to build tools that monitor its application across companies – leave alone countries.

5. Can you offer or trade computing capacity? If you know how much you need, when and where, why not “sell” spare capacity? Again, S+S comes into play here. Grid computing is possibly the best example of implementation of a similar concept today.

3. What tools & techniques are available?

Hyper-V sounds like an obvious answer, but we are at risk of sounding like the proverbial person with just 1 hammer in the toolbox, to whom everything looks like a nail.

Virtualization is one powerful tool, but it must be used appropriately. One must carefully choose which workloads to virtualize, then which of those virtualized workloads can be combined on a single physical tier. Again, given a workload profile, that physical tier may look entirely different from your current one. Also, most often we speak only of host virtualization. For a complete solution, we must find the best combination of host, storage and network virtualization.

A caveat to keep in mind is that virtualization may be self-defeating without proper management practices. The ease of deploying virtual machines may lead operators to spawn far more than necessary. I have seen a few examples of this in large deployments.

Regulation (in the form of prescriptive guidance) may address some of the problem, but charging money is more effective. The idea of trading computing power may become useful in this scenario: imagine that you planned and budgeted for 200 VMs, but find out that you’re running just 150. You could sell the capacity for the remaining 50 to another part of your organization that requires it. They wouldn’t even need to buy or host servers. Who said that market economy principles cannot be applied to IT governance?

Co-hosting is another technique to optimize resource consumption, often neglected on Windows. If you can virtualize two workloads and run them together without significant impacts in performance, you may be able to gain even more by running them on the same o/s instance. The applications must of course be compatible (able to coexist). Thus, you eliminate the overhead of virtualization. Tools like WSRM allow you to change resource allocation dynamically, adapting to workload requirements. Unix and Mainframes have been doing this for decades, along with virtualization.

IIS6 and 7, for instance, are classical examples where co-hosting of several websites works very well. SQL2005 and 2008 are good examples too, where you can co-host several databases in one instance and several instances on one machine.

As for capacity optimization tools, I could not find a silver bullet. I mentioned a few so far; here’s a quick summary:

- System Center Virtual Machine Manager, with its workload analysis and placement functions, is instrumental in devising the best resource allocation.

- The Microsoft Assessment and Planning Toolkit is a useful, free instrument to plan for virtualization (amongst other things).

- System Center Capacity Planner is also very useful in designing the target architecture for certain workloads (Exchange, Sharepoint, Operations Manager).

- For a far more sophisticated (and expensive) capacity management and planning suite, you may want to look at tools like SAS.

- System Center Operations Manager, plus management packs provided by OEMs, is useful to obtain a baseline of resource utilization.

- IBM’s Active Energy Manager is a great example of what we can do with the data.

4. Further reading

Here are a few pointers that may help inform a discussion:

- Lewis Curtis’s blog: http://blogs.technet.com/lcurtis/

- Little Miss Enviro-Geek http://blogs.technet.com/lmeg/default.aspx

- The Green Datacenter Blog: http://www.greenm3.com/2008/07/new-coal-electr.html

- MAP Toolkit

- Microsoft’s Environment web page

- IBM Green Datacenter paper

- IBM Active Energy Manager

- Windows Server 2008 Power Savings

- Green Computing Paper

- The Green Grid

- Infrastructure Planning and Design

Hyper-V and Server 2008 RTM

gmarchetti — Thu, 21 Feb 2008 20:18:00 GMT

With general availability of Windows Server 2008 Microsoft ships the beta 1 release of Windows Virtualization or Hyper-V (aka. Viridian).

Customers, partners, developers, IT Professionals, everyone buying a copy of Windows Server 2008 will have the opportunity to test drive the Microsoft hypervisor Hyper-V.

The version of Hyper-V included in the RTM build of Windows Server 2008 is identical to the build made available as part of the RC1 release of the server OS. The final release of Hyper-V will be available with 180 days from now. Microsoft’s hypervisor will only be available for x64 hardware platforms supporting hardware virtualization (Intel VT or AMD-V technologies). It will not be available for IA64 (Itanium) platforms. See resources for more information or here for general information about virtualization.

Hyper-V is still in beta. Support for Hyper-V is only available via online forums and newsgroups. See resources section.

The beta release of Hyper-V has limited guest operating support:

Windows Server 2008 32-bit and 64-bit (x64)

Windows Server 2003 32-bit and 64-bit (x64)

SUSE LINUX Enterprise Server 10 with SP1

Other OS installations might run as well but are not “supported” in the beta. The full list of supported operating systems in will be announced prior to RTM.

Virtualization solutions from Microsoft are:

Presentation Virtualization via Terminal Services

Application Virtualization via SoftGrid

Virtual PC

Virtual Server

Hyper-V Server Virtualization

Note:

The Virtual Server product will continue to be available. Hyper-V is only supported on 64-bit platforms, Virtual Server is the Microsoft offering for 32-bit platforms and other systems not providing the necessary hardware infrastructure for Hyper-V.

IT PROs

How does the general availability Hyper-V beta impact IT PROs? For some of the partners this is the first time they will be exposed to Hyper-V. They may be experienced with Virtual Server and/or Virtual PC. Hyper-V is a novel concept: a layer between the hardware and a variety of operating systems handling scheduling and memory allocation (amongst other things). It is the foundation of a dynamic infrastructure, making resource optimization and re-configuration far simpler than before and simplifying operations. Depending on the type of application there might be areas where its performance might be impacted. Heavily I/O bound and graphics-intensive applications are among those. Deciding where, what and how to virtualize infrastructure requires careful thought. Our Infrastructure Planning and Design series of documents will help guide that process.

Resources

Some resources you might find useful:

Forums

Microsoft SoftGrid Application Virtualization
System Center Virtual Machine Manager

Server Virtualization

Documentation

Technet Center for Virtualization

API

WMI

Hyper-V WMI API

HyperCall API

Products