HPC, Virtualization and Random Thoughts : Virtualization

Live Migration, Cluster Shared Volumes & Networks

gmarchetti — Fri, 02 Oct 2009 02:42:49 GMT

The recommendation for people setting up live migration clusters is to isolate different kinds of traffic on their own networks:

Public network to access the cluster and the virtual machines running on it
“Private” cluster heartbeat network
“live migration” network
iSCSI network, if required to access shared storage

How do we determine what traffic goes where?

For public and private, the failover cluster manager tool is self-explanatory:

We select the appropriate cluster network properties. If we want to limit such network to private traffic, we do not allow clients to connect through it.

If we don’t want the cluster to use such network at all, e.g. because it is dedicated to iSCSI, we select the “Do not allow…” button.

How about the live migration traffic, though? It can be quite heavy, as we are copying memory pages from one host to another. We can select in which order to use cluster networks for such traffic through the failover cluster manager

The property requires some digging: expand “services and applications”, select the virtual machine in question, then in the main panel right-click on “virtual machine <name>” and you’ll see tab called “network for live migration”. You can then select and sort in order of priority the networks that you want to use. By default, live migration will select a network that is NOT used for CSV traffic. Note that you may have networks in this panel that were not selected for cluster use before. If you use iSCSI, de-select the relevant entry to make sure that the live migration traffic does not go through that network.

This brings me to cluster shared volumes. One of the great features of CSVs is that if the storage link (iSCSI, fibre) becomes unavailable for any reason on a node, storage traffic can be redirected over the cluster network to another node and hence to the storage device. But which cluster network?

Inter-node communications and CSV traffic will use the available network authorized for cluster use that has the lowest metric value. We can see the metrics with old cluster.exe

C:\Windows\system32>cluster net /prop
Listing properties for all networks:

T Network              Name                     Value
-- -------------------- ------------------------ -----------------------
SR Cluster Network 1    Name                     Cluster Network 1
MR Cluster Network 1    IPv6Addresses            <cut on purpose>
MR Cluster Network 1    IPv6PrefixLengths        <..>
MR Cluster Network 1    IPv4Addresses            <cut on purpose>
MR Cluster Network 1    IPv4PrefixLengths        <..>
SR Cluster Network 1    Address                  <..>
SR Cluster Network 1    AddressMask              <..>
S Cluster Network 1    Description
D Cluster Network 1    Role                     3 (0x3)
D Cluster Network 1    Metric                   10001 (0x2711)
D Cluster Network 1    AutoMetric               0 (0x0)
SR Cluster Network 2    Name                     Cluster Network 2
MR Cluster Network 2    IPv6Addresses
MR Cluster Network 2    IPv6PrefixLengths
MR Cluster Network 2    IPv4Addresses            <..>
MR Cluster Network 2    IPv4PrefixLengths        <..>
SR Cluster Network 2    Address                  <..>
SR Cluster Network 2    AddressMask              <..>
S Cluster Network 2    Description
D Cluster Network 2    Role                     1 (0x1)
D Cluster Network 2    Metric                   1000 (0x3e8)
D Cluster Network 2    AutoMetric               1 (0x1)

Note the 3 values:

Role: 1 for a private network, 0 for ignored by cluster, 3 for mixed traffic
Metric: the “weight” of the connection, generally in the 10,000 range for public networks, 1,000 for private ones. If a network has a default gateway, it is considered public; if not, private. Should there be more than one private or public network, the metric is incremented by 100 in order of enumeration (e.g. private network 2 will have a default metric of 1,100)
Autometric: 1 if the metric is set automatically by the cluster, 0 if you have set it manually.

So in my simple case the heartbeat network will also be used for CSV traffic. If you have more than 1 private network and you want to prioritize them, you can set the metric with cluster.exe, e.g.

C:\Windows\system32>cluster net "Cluster Network 2" /prop metric=1001

C:\Windows\system32>cluster net "Cluster Network 2" /prop

Listing properties for 'Cluster Network 2':

T Network              Name                     Value
-- -------------------- ------------------------ -----------------
SR Cluster Network 2    Name                     Cluster Network 2
MR Cluster Network 2    IPv6Addresses
MR Cluster Network 2    IPv6PrefixLengths
MR Cluster Network 2    IPv4Addresses                 <..>
MR Cluster Network 2    IPv4PrefixLengths             <..>
SR Cluster Network 2    Address                       <..>
SR Cluster Network 2    AddressMask                   <..>
S Cluster Network 2    Description
D Cluster Network 2    Role                     1 (0x1)
D Cluster Network 2    Metric                   1001 (0x3e9)
D Cluster Network 2    AutoMetric               0 (0x0)

Redirection of the traffic is automatic: if a network becomes unavailable, the next-lowest-metric one will be used. If another network with a lower metric becomes available, it will be used from that point onwards.

In Summary

By default, live migration traffic will be put on the network with the second-lowest metric. CSV traffic will be put on the the network with the lowest metric. In this simple example, I just have a public and private network, so the public one is used for live migration and the private one for csv and cluster traffic.

P2V with SCVMM – a quick tip

gmarchetti — Wed, 22 Jul 2009 00:54:53 GMT

System Center Virtual Machine Manager (SCVMM) has been offering a relatively simple way of doing physical-to-virtual migrations (P2V) for a while. You just click on the “Convert Physical Server” icon and off you go. Despite the name, it also works with client target machines. It’s simple, if you do some preparation work before.

In fact, VMM will ask you for name or ip address of the machine in question and for administrator credentials on it. Those will be used to reach the machine and install a p2v agent on it. For the process to work correctly, you must let through the firewall of the target machine:

WMI traffic
http
file and print
remote management

Also, make sure that the ADMIN$ share exists and start the Windows remote management service on the target machine.

By default, most of these ports and services are closed.

VHD Boot

gmarchetti — Wed, 15 Jul 2009 01:35:30 GMT

With Windows 7 and Server 2008 R2 you get the opportunity to boot directly from a vhd file. The operating system in the vhd file will have direct access to the machine hardware. It will not run as a virtual machine with synthetic or emulated adapters, but as a “real machine”. VHD happens to be the format that is used to represent a disk to the o/s. The physical disk will contain a set of vhd files and still be visible as a disk to the o/s you boot. Thus, you won’t require a partition per o/s.

Assuming that you are running Windows 7 or 2008 R2, here’s how you can set it up:

1. Create a vhd file to contain your o/s.

I found that 15 GB are enough for Server 2008 R2 + Hyper-V role (you can enable any role on the o/s in that VHD). You can use diskpart from the command line or the disk management tool. Make sure to select a fixed disk size.

Mount that vhd file, e.g. to drive letter W:

2. Apply a WIM image to the VHD file you just created.

You can generate a WIM image of a pre-installed “golden” machine with the imagex tool, part of the Windows Automated Installation Kit (WAIK)

You can use the WIM image provided with the Windows installation media in sources\install.wim

If all you have is an iso file (e.g. a 2008 R2 evaluation version you’ve just downloaded), there are utilities like MagicISO which will let you mount it as a disk, so you can use the install.wim within the file.

The simplest way to apply the wim image is to use the Install-Windowsimage powershell script, which you’ll find on MSDN. The installation media contains several versions of Windows, so make sure you select the one you are licensed for.

In powershell, type:

.\Install-WindowsImage.ps1 -WIM D:\Sources\Install.wim

to obtain a list of the available images on your installation DVD (D:\). Note the index number of the image you are interested in.

Type

.\Install-WindowsImage.ps1 -WIM D:\Sources\Install.wim -Apply -Index 3 -Destination W:\

to apply the 3rd image on the VHD drive you mounted previously (W:\)

3. Make the VHD bootable

Open a command prompt and type:

W:\windows\system32\bcdboot w:\windows

Bcdboot creates the boot control data (bcd) block to boot Windows from the vhd file and sets it as default option.

4. Check the bcd entry and set your preferred default

At an administrator’s command prompt, type:

bcdedit /v

Note the identifier of the entry that you want to use as default boot option.

Type:

bcdedit /default {identifier}

to set the default.

That’s it – you can now boot from your vhd file.

You need not stop here, however: there is no need to start from a regular o/s installation: you can configure vhd boot from the installation media and have all your Windows o/s boot from vhd files. Keith Combs explains how on his blog.

A Free Book on Microsoft Virtualization

gmarchetti — Fri, 13 Mar 2009 02:01:59 GMT

Understanding Microsoft Virtualization Solutions from Microsoft Press is available as a FREE download.

This 15MB E-Book gives an overview of all current Microsoft Virtualization technologies: Hyper-V, the Microsoft Enterprise Desktop Virtualization (MED-V), and VDI. It also describes which management solutions are available for them (e.g. System Center Virtual Machine Manager) and how they fit together. It is worth reading when planning the virtualization of your infrastructure.

You can find it here: http://csna01.libredigital.com/?urmvs17u33

Turning hyper-v on and off

gmarchetti — Mon, 08 Dec 2008 00:19:00 GMT

I use hyper-v on my laptop. When I know I don't need VMs for the day, I can squeeze a bit more performance out of the machine by turning hyper-v off with:

bcdedit /set hypervisorlaunchtype off

and a reboot. To turn it back on:

bcdedit /set hypervisorlaunchtype on (or auto start)

and reboot.

Stop Climate Change?! – Part 2

gmarchetti — Tue, 22 Jul 2008 03:41:00 GMT

I have been investigating some more in the area of Green IT, S+S. Some ideas and a lot of questions have come to mind. Please read on and let me know if they make any sense.

By the way, Part 1 is here :-)

1. How do you understand the status quo?

This may prove to be the most difficult part of the job. There aren’t many tools available. System Center Operations Manager, plus a few OEM management packs, are a starting point. Alas, you must build your own model to establish correlations between power utilization measured and applications over time. From those, you can derive a measure of efficiency. A few 3^rd-party applications (e.g. Verdiem’s Surveyor, Avocent and APC InfrastruXure) do a better job of establishing the baseline, although again they do little for the correlation analysis.

IBM’s Active Energy Manager goes a step further (on IBM hardware) by allowing you study trends and to take action on specific energy-related conditions. Again, it is not a complete “IT intelligence” tool.

2. How do you design your infrastructure and applications to optimize consumption?

Once you understand what type of load consumes what power (no small feat):

1. Can you reduce the physical tiers of your architecture? For instance, if you have a memory-intensive application and a CPU-intensive one, you may want to co-host them, thus using all the available cores and saving a few machines’ worth of power. This will only work from a performance point of view if you manage resource allocation tightly to avoid contention. In our example, you would run a thread belonging to the memory intensive application on 1 core and a thread of the cpu-intensive one on the other core of the same CPU socket. Before embarking on such a consolidation exercise, you will want to estimate the costs and the savings, in terms of power and money. Also keep in mind that as a consequence of the changed workload, you may require different hardware (e.g. “whole machines” rather than just blades) to optimize your power consumption profile over time.

2. Can you reduce the logical tiers of your architecture? Here’s an example: your application may use Sharepoint as a front-end, windows workflow to manage business logic, SQL for data processing, all running on separate hardware. Sharepoint can host workflows. SQL handles workflows in Integration Services and it can host an in-process CLR. With some clever re-architecting of your application, you may be able to get rid of the middle tier by using some combination of the two workflow services. The whole area of “power-conscious” applications is yet to be explored. We’re investigating.

3. Can you offload a tier of your architecture? Here’s where Software + Services comes into play. For instance, you may consider using an on-line storage service (e.g. SQL Server Data Services, aka CloudDB or Sitka) instead of hosting your own SQL. If you have a compute-intensive application, you may want to farm it out to a HPC provider and pay by CPU cycles utilized (Microsoft will offer such a service, now in pilot stage with a few ISVs). If your provider is able to consolidate several users’ workloads on its servers and charge for capacity consumed, the overall carbon footprint may be reduced – along with your costs.

4. If you do offload a function, how do you measure its performance against SLAs? This is actually the most difficult point. Technology is available to do all of the above (although not necessarily on Windows). Capacity-on-demand, for instance, has been a feature of certain Mainframes and Unix systems for years. Hosted services offering are widely available. However, different security boundaries and political pressures make it difficult to build tools that monitor its application across companies – leave alone countries.

5. Can you offer or trade computing capacity? If you know how much you need, when and where, why not “sell” spare capacity? Again, S+S comes into play here. Grid computing is possibly the best example of implementation of a similar concept today.

3. What tools & techniques are available?

Hyper-V sounds like an obvious answer, but we are at risk of sounding like the proverbial person with just 1 hammer in the toolbox, to whom everything looks like a nail.

Virtualization is one powerful tool, but it must be used appropriately. One must carefully choose which workloads to virtualize, then which of those virtualized workloads can be combined on a single physical tier. Again, given a workload profile, that physical tier may look entirely different from your current one. Also, most often we speak only of host virtualization. For a complete solution, we must find the best combination of host, storage and network virtualization.

A caveat to keep in mind is that virtualization may be self-defeating without proper management practices. The ease of deploying virtual machines may lead operators to spawn far more than necessary. I have seen a few examples of this in large deployments.

Regulation (in the form of prescriptive guidance) may address some of the problem, but charging money is more effective. The idea of trading computing power may become useful in this scenario: imagine that you planned and budgeted for 200 VMs, but find out that you’re running just 150. You could sell the capacity for the remaining 50 to another part of your organization that requires it. They wouldn’t even need to buy or host servers. Who said that market economy principles cannot be applied to IT governance?

Co-hosting is another technique to optimize resource consumption, often neglected on Windows. If you can virtualize two workloads and run them together without significant impacts in performance, you may be able to gain even more by running them on the same o/s instance. The applications must of course be compatible (able to coexist). Thus, you eliminate the overhead of virtualization. Tools like WSRM allow you to change resource allocation dynamically, adapting to workload requirements. Unix and Mainframes have been doing this for decades, along with virtualization.

IIS6 and 7, for instance, are classical examples where co-hosting of several websites works very well. SQL2005 and 2008 are good examples too, where you can co-host several databases in one instance and several instances on one machine.

As for capacity optimization tools, I could not find a silver bullet. I mentioned a few so far; here’s a quick summary:

- System Center Virtual Machine Manager, with its workload analysis and placement functions, is instrumental in devising the best resource allocation.

- The Microsoft Assessment and Planning Toolkit is a useful, free instrument to plan for virtualization (amongst other things).

- System Center Capacity Planner is also very useful in designing the target architecture for certain workloads (Exchange, Sharepoint, Operations Manager).

- For a far more sophisticated (and expensive) capacity management and planning suite, you may want to look at tools like SAS.

- System Center Operations Manager, plus management packs provided by OEMs, is useful to obtain a baseline of resource utilization.

- IBM’s Active Energy Manager is a great example of what we can do with the data.

4. Further reading

Here are a few pointers that may help inform a discussion:

- Lewis Curtis’s blog: http://blogs.technet.com/lcurtis/

- Little Miss Enviro-Geek http://blogs.technet.com/lmeg/default.aspx

- The Green Datacenter Blog: http://www.greenm3.com/2008/07/new-coal-electr.html

- MAP Toolkit

- Microsoft’s Environment web page

- IBM Green Datacenter paper

- IBM Active Energy Manager

- Windows Server 2008 Power Savings

- Green Computing Paper

- The Green Grid

- Infrastructure Planning and Design

A Hybrid OS Cluster Solution

gmarchetti — Tue, 03 Jun 2008 22:28:09 GMT

Thomas Varlet, of Microsoft France, and Dr. Patrice Calegari, of BULL SAS, have written an excellent paper on how to build hybrid clusters, i.e. clusters where 2 or more operating systems can be run at the same time. It is recommended reading, in my opinion, for those of us who use both Linux and Windows HPC solutions. You'll find the paper here.

Hyper-V and Server 2008 RTM

gmarchetti — Thu, 21 Feb 2008 20:18:00 GMT

With general availability of Windows Server 2008 Microsoft ships the beta 1 release of Windows Virtualization or Hyper-V (aka. Viridian).

Customers, partners, developers, IT Professionals, everyone buying a copy of Windows Server 2008 will have the opportunity to test drive the Microsoft hypervisor Hyper-V.

The version of Hyper-V included in the RTM build of Windows Server 2008 is identical to the build made available as part of the RC1 release of the server OS. The final release of Hyper-V will be available with 180 days from now. Microsoft’s hypervisor will only be available for x64 hardware platforms supporting hardware virtualization (Intel VT or AMD-V technologies). It will not be available for IA64 (Itanium) platforms. See resources for more information or here for general information about virtualization.

Hyper-V is still in beta. Support for Hyper-V is only available via online forums and newsgroups. See resources section.

The beta release of Hyper-V has limited guest operating support:

Windows Server 2008 32-bit and 64-bit (x64)

Windows Server 2003 32-bit and 64-bit (x64)

SUSE LINUX Enterprise Server 10 with SP1

Other OS installations might run as well but are not “supported” in the beta. The full list of supported operating systems in will be announced prior to RTM.

Virtualization solutions from Microsoft are:

Presentation Virtualization via Terminal Services

Application Virtualization via SoftGrid

Virtual PC

Virtual Server

Hyper-V Server Virtualization

Note:

The Virtual Server product will continue to be available. Hyper-V is only supported on 64-bit platforms, Virtual Server is the Microsoft offering for 32-bit platforms and other systems not providing the necessary hardware infrastructure for Hyper-V.

IT PROs

How does the general availability Hyper-V beta impact IT PROs? For some of the partners this is the first time they will be exposed to Hyper-V. They may be experienced with Virtual Server and/or Virtual PC. Hyper-V is a novel concept: a layer between the hardware and a variety of operating systems handling scheduling and memory allocation (amongst other things). It is the foundation of a dynamic infrastructure, making resource optimization and re-configuration far simpler than before and simplifying operations. Depending on the type of application there might be areas where its performance might be impacted. Heavily I/O bound and graphics-intensive applications are among those. Deciding where, what and how to virtualize infrastructure requires careful thought. Our Infrastructure Planning and Design series of documents will help guide that process.

Resources

Some resources you might find useful:

Forums

Microsoft SoftGrid Application Virtualization
System Center Virtual Machine Manager

Server Virtualization

Documentation

Technet Center for Virtualization

API

WMI

Hyper-V WMI API

HyperCall API

Products

Hyper-V
Virtual Server
Virtual PC
Terminal Services
SoftGrid

Programs

VHD Test Drive

Blogs

Virtualization team blog

Virtual PC Guy

John Howard

Hardware

Intel VT (aka. Vanderpool)

AMD-V (aka. Pacifica)

Miscellaneous

Virtualization Homepage

Clustering Hyper-V Screencast

Hyper-V

Computing Utilities

gmarchetti — Fri, 10 Aug 2007 19:46:00 GMT

So, imagine this scenario: you have a rack full of servers that hum along at 20% of capacity, performing "mundane" tasks. Wouldn't it be nice to harness at least part of that remaining computing power for your batch calculations? Wouldn't it be nice to save some money by not having to expand that costly, fast cluster you use for your fluid-dynamic / atom-splitting / weather-forecasting computations? How about using the same programming model, job submission tools and management tools that you use for your main cluster on those lazy machines? How about having the same security and isolation?

Here's the idea, admittedly far-fetched (well give us some time):

- Virtualize the "lazy" machines with physical-to-virtual tools, like system center virtual machine manager (http://www.microsoft.com/systemcenter/scvmm/default.mspx)

- Install a hypervisor on those machines. We are producing one for Windows Server 2008, in case you haven't noticed :-). Take a look at Arlindo's blog on http://blogs.technet.com/aralves/archive/2007/02/28/longhorn-hypervisor-demo.aspx for an idea of what is possible.

- Configure a partition (virtual machine) that does the mundane job. It contains the same n. of virtual processors as in the real machine (or up to what is supported by the hypervisor, whatever is smaller), but it is assigned a portion of the memory and the ability to consume up to 20% of CPU (or whatever is appropriate). System Center will help you with sizing.

- Configure a second partition with Compute Cluster Server and make it a member of your cluster. Again, this has as many virtual processors as the real machine (or up to the supported limit), but it is allowed to use up to the remaining computing power minus about 20% (so, 60% in our example. The 20% is a rule of thumb for the hypervisor overhead, not a scientific recommendation).

- if your workload requirements change, you can dynamically alter the CPU allocation to the partitions. E.g. if you need 60% of capacity for the "mundane" tasks at certain times, you can schedule a script that pauses the cluster nodes, then changes the allocation. There is no need to switch on or off and repurpose machines.

What are the scenarios where this is suitable?

Virtualization does impose a performance penalty, even with hypervisors. Also, some hardware (e.g. infiniband) will not be virtualized. So, don't use this technique for real-time, massively parallel computations.

However, for parametric sweeps, especially in batch jobs, this recoups some valuable computing power with little management overhead. Having 1000 CPUs, even if they are in reality a time-share of real ones, may be a worthy addition to 100 dedicated ones.

So, worth a try? If you think so, drop me a note on gmarchet@microsoft.com