HPC, Virtualization and Random Thoughts : Windows Server 2008

P2V with SCVMM – a quick tip

gmarchetti — Wed, 22 Jul 2009 00:54:53 GMT

System Center Virtual Machine Manager (SCVMM) has been offering a relatively simple way of doing physical-to-virtual migrations (P2V) for a while. You just click on the “Convert Physical Server” icon and off you go. Despite the name, it also works with client target machines. It’s simple, if you do some preparation work before.

In fact, VMM will ask you for name or ip address of the machine in question and for administrator credentials on it. Those will be used to reach the machine and install a p2v agent on it. For the process to work correctly, you must let through the firewall of the target machine:

WMI traffic
http
file and print
remote management

Also, make sure that the ADMIN$ share exists and start the Windows remote management service on the target machine.

By default, most of these ports and services are closed.

Turning hyper-v on and off

gmarchetti — Mon, 08 Dec 2008 00:19:00 GMT

I use hyper-v on my laptop. When I know I don't need VMs for the day, I can squeeze a bit more performance out of the machine by turning hyper-v off with:

bcdedit /set hypervisorlaunchtype off

and a reboot. To turn it back on:

bcdedit /set hypervisorlaunchtype on (or auto start)

and reboot.

What is new in virtualization with Windows Server 2008 R2?

gmarchetti — Thu, 13 Nov 2008 01:43:00 GMT

There are some quite interesting improvements in Windows Server 2008 R2 (what was wrong with W7 as a name?) that help us progress toward a dynamic infrastructure. Three of them are worthy of highlighting: live migration of virtual machines in hyper-v, cluster shared volumes and core parking.

1. Live Migration

Live migration refers to the ability of moving a running virtual machine from one host server to another without loss of service. For this to happen, we have to transfer the current virtual machine state and memory pages between machines and we have to warrant both servers the same level of access to the virtual machine files. The process can be summarized as follows:

Create a virtual machine on the target server
Copy the memory pages of the running virtual machine in question from the source to the target server via Ethernet. While we copy, those memory pages may change, so after an initial pass we have to go back and copy the changed set again, until a minimum threshold number of pages is reached. It is hard to fix the threshold: ideally, it will be the number of pages that can be copied within a TCP connection timeout, so the clients won’t notice.
Pause the source machine; copy its state across.
Resume the target machine, issue ARP command to update routing tables.

For (3) to happen quickly and transparently to the clients, the target server must have immediate access to the virtual machine files. It cannot wait for a disk volume to fail-over and possibly go through file system checks. That’s where cluster shared volumes come in.

2. Cluster Shared Volumes

Cluster Shared Volumes enable concurrent access to the same LUN by several nodes. Consequently, all the nodes see the same NTFS file-system and namespace. By the way, CSV is not a parallel or a cluster file system. It was designed with the live migration scenario in mind.

Since the host servers already mount the CSV, there is no need to arbitrate for disk access and fail over the volume hosting the virtual machine files. All you need to do is transfer ownership of those files and their locks to the target server.

CSVs are implemented via a filter driver mechanism, which is used to establish the access path to the underlying LUNs. This also enhances our fail-over ability, as file system requests will be redirected over the network to another server if a direct SAN access is no longer available.

3. Core idling or parking

Changes in Windows 7 power management allow for “density” scheduling, i.e. minimizing the number of processor cores on which work is done, hence maximizing their utilization. The idle cores can be put to sleep (low-power state Cx under the ACPI specifications), thus reducing power consumption. Hyper-V can take advantage of this feature and schedule its virtual machines accordingly. Power management policies can be controlled via WMI, policies and scripts.

If you combine “density” scheduling with the ability to move virtual machines among hosts, you achieve quite a scalable, efficient and dynamic solution to the distributed resource allocation problem. Now, all that remains to do is automate it. Stay tuned.

4. References

ACPI explanation on Wikipedia

WinHEC 2008 conference whitepapers

Engineering Windows 7 blog

The Windows blog

The Windows Server 2008 R2 Reviewers' Guide http://www.microsoft.com/windowsserver2008/en/us/r2.aspx

Upgrading from an evaluation version

gmarchetti — Tue, 04 Nov 2008 02:39:00 GMT

I have received a few questions about upgrades from the evaluation version that you can download from microsoft.com/hpc to a full version.

The good news is that the evaluation version is fully functional, so you won't need a complete re-installation. The only thing you need to do is obtain a full licence key, then:

- To upgrade the hpc pack tools you have to run “upgrade.exe” on the head node. The hpc pack CD contains the upgrade.exe file.

- To upgrade the o/s, you have to obtain a full licence key for all the nodes, then run slmgr.vbs –ipk <new licence> across the cluster. You can do that from the command line (clusrun /all) or via the GUI.

You can also use slmgr.vbs to extend the evaluation period by another 60 days. When you are approaching the end of the evaluation, simply run slmgr.vbs -rearm across the cluster. Note that the evaluation does not require activation, but a full licence does.

Please see http://support.microsoft.com/kb/948472 for more information.

A Hybrid OS Cluster Solution

gmarchetti — Tue, 03 Jun 2008 22:28:09 GMT

Thomas Varlet, of Microsoft France, and Dr. Patrice Calegari, of BULL SAS, have written an excellent paper on how to build hybrid clusters, i.e. clusters where 2 or more operating systems can be run at the same time. It is recommended reading, in my opinion, for those of us who use both Linux and Windows HPC solutions. You'll find the paper here.

Infiniband on HPC Server 2008 - again

gmarchetti — Tue, 18 Mar 2008 21:52:00 GMT

After I published a document about the installation of Mellanox Infiniband on Server 2008, I have received some good feedback that deserves sharing. Note that:

The WinIB 1.4 beta available for download today on Mellanox's web site does not work with the HPC Server 2008 March CTP. We are working with Mellanox to fix that.

The procedure I illustrated in that document uses a "trick": Deploy the Mellanox package with msiexec first, add the Infiniband network to the cluster configuration later. This works both with our deployment tools and with 3rd parties'. However, one can exploit the built-in HPC Server 2008 tools better. Here's how:

1. Install the Mellanox WinIB package (currently WinIB_x86_1_4_0_2094.msi) on the head node. Set up the cluster network configuration to include Infiniband as the MPI network.

2. Create an o/s image and deployment template for the compute nodes.

3. In the Admin Console, right-click on the image and select Manage Drivers. You need 3 drivers for the card to be visible in the admin console, hence configurable:

ib_bus.inf Mellanox InfiniBand Fabric driver
mthca.inf InfiniBand Host Channel Adapter driver
netipoib.inf Mellanox IP over Infiniband protocol driver

You will find the first 2 files in the C:\Program Files\Mellanox\WinIB\Drivers on the head node after installation of the WinIB package. The last one will be in C:\Program Files\Mellanox\WinIB\IPoIB

4. Copy WinIB_x86_1_4_0_2094.msi package to %CCP_DATA%\InstallShare on the head node.

5. Edit the compute nodes deployment template and add an Installation->Unicast Copy from z:\<winib package>.msi to c:\<winib package>.msi; move the copy operation before the "Install CCP" task.

6. In the same template, add an Installation->Execute OS command: msiexec /i c:\<winib package>.msi /qn ADDLOCAL=ALL.

7. Deploy the compute nodes.

8. When they are deployed, you can start the Network Direct provider with clusrun /nodes:<list of nodes> "%WinIB_HOME%\IPoIB\NDI\ndinstall.exe" -i

HPC Server 2008 webcasts - again!

gmarchetti — Tue, 18 Mar 2008 01:37:00 GMT

Hello again,

I'll be doing a series of webcasts soon, hopefully on the feature-complete beta 2 of HPC Server 2008. They will be mostly demonstrations, with a few slides for those concepts that are not evident in the software. Here is the schedule (all times are PST):

- 5/9/2008 08:00 AM: Windows HPC Server 2008: Management and Diagnostics in High-Performance Computing (Level 200)

- 5/23/2008 08:00 AM: Windows HPC Server 2008: High Availability and Diagnostics for High-Performance Computing (Level 200)

- 5/30/2008 09:30 AM: Windows HPC Server 2008: Job Scheduler and SOA in High-Performance Computing (Level 200)

Phil Pennington will also present on current efforts to develop a unified parallel programming model.

- 5/2/2008 08:00 AM: Future of Multi/Many-Core and the Convergence of Client and Cluster in Parallel Computing (Level 300)

Please click on the links to register.

Mellanox Infiniband on HPC Server 2008

gmarchetti — Fri, 29 Feb 2008 04:13:41 GMT

One would assume that Infiniband on Windows is just going to be as easy as any other plug and play device installation. Well, in some cases it is. When you have some old cards and an old switch, no documentation, both not supported any longer and in an unknown state, it may not be! I am sure this experience is rather common, so I’ve decided to document what I did to make them work. In this case, I will focus on the Mellanox Infiniband stack, as it is the only one I could find that has public beta support for HPC Server 2008. Besides, Mellanox is the dominant provider of Infiniband hardware, even if it is then re-branded and re-sold by others.

The document is a bit long for a blog post, so I have put it on Windowshpc.net

DISCLAIMER: This is not official guidance, just my notes. No guarantee is offered or implied. Your experience may differ, mileage may vary etc… etc… Also, I do not claim to be an Infiniband expert in any way. If you have comments, corrections, suggestions to make, please do so by responding to this post. I will update the document accordingly.

Tags: Infiniband, HPC, Windows Server 2008

Powered by Qumana

Hyper-V and Server 2008 RTM

gmarchetti — Thu, 21 Feb 2008 20:18:00 GMT

With general availability of Windows Server 2008 Microsoft ships the beta 1 release of Windows Virtualization or Hyper-V (aka. Viridian).

Customers, partners, developers, IT Professionals, everyone buying a copy of Windows Server 2008 will have the opportunity to test drive the Microsoft hypervisor Hyper-V.

The version of Hyper-V included in the RTM build of Windows Server 2008 is identical to the build made available as part of the RC1 release of the server OS. The final release of Hyper-V will be available with 180 days from now. Microsoft’s hypervisor will only be available for x64 hardware platforms supporting hardware virtualization (Intel VT or AMD-V technologies). It will not be available for IA64 (Itanium) platforms. See resources for more information or here for general information about virtualization.

Hyper-V is still in beta. Support for Hyper-V is only available via online forums and newsgroups. See resources section.

The beta release of Hyper-V has limited guest operating support:

Windows Server 2008 32-bit and 64-bit (x64)

Windows Server 2003 32-bit and 64-bit (x64)

SUSE LINUX Enterprise Server 10 with SP1

Other OS installations might run as well but are not “supported” in the beta. The full list of supported operating systems in will be announced prior to RTM.

Virtualization solutions from Microsoft are:

Presentation Virtualization via Terminal Services

Application Virtualization via SoftGrid

Virtual PC

Virtual Server

Hyper-V Server Virtualization

Note:

The Virtual Server product will continue to be available. Hyper-V is only supported on 64-bit platforms, Virtual Server is the Microsoft offering for 32-bit platforms and other systems not providing the necessary hardware infrastructure for Hyper-V.

IT PROs

How does the general availability Hyper-V beta impact IT PROs? For some of the partners this is the first time they will be exposed to Hyper-V. They may be experienced with Virtual Server and/or Virtual PC. Hyper-V is a novel concept: a layer between the hardware and a variety of operating systems handling scheduling and memory allocation (amongst other things). It is the foundation of a dynamic infrastructure, making resource optimization and re-configuration far simpler than before and simplifying operations. Depending on the type of application there might be areas where its performance might be impacted. Heavily I/O bound and graphics-intensive applications are among those. Deciding where, what and how to virtualize infrastructure requires careful thought. Our Infrastructure Planning and Design series of documents will help guide that process.

Resources

Some resources you might find useful:

Forums

Microsoft SoftGrid Application Virtualization
System Center Virtual Machine Manager

Server Virtualization

Documentation

Technet Center for Virtualization

API

WMI

Hyper-V WMI API

HyperCall API

Products

Hyper-V
Virtual Server
Virtual PC
Terminal Services
SoftGrid

Programs

VHD Test Drive

Blogs

Virtualization team blog

Virtual PC Guy

John Howard

Hardware

Intel VT (aka. Vanderpool)

AMD-V (aka. Pacifica)

Miscellaneous

Virtualization Homepage

Clustering Hyper-V Screencast

Hyper-V

Windows Server 2008 High-Availability Clusters

gmarchetti — Wed, 23 Jan 2008 04:32:00 GMT

It is no secret that HPC Server 2008 will offer the option to make the head node of a HPC cluster highly available. This feature is not in beta 1, but it is being developed. It will exploit fail-over mechanisms provided by Server 2008 (enterprise edition or better), so I thought I'd mention some highlights in this area too.

High-availability clusters are difficult to set up and troubleshoot on several platforms. With Windows Server 2003 we made progress in simplifying them, but limitations are still significant:

You need a configuration that is fully and specifically certified as a cluster in order to obtain support when things go wrong.
There is very limited support for geo-clusters, because of limitations in intra-cluster communications, no awareness of storage location and cluster quorum models. Also, geo-clusters require yet another level of certification.
Writing cluster-aware applications is not easy. It requires knowledge of cluster-specific APIs in order to produce “resources” usable by the cluster software. Scripting generic application fail-over is supported, but limited in functionality.
Troubleshooting by reading cluster logs requires very deep knowledge to interpret the cryptic messages therein.

These limitations may hamper adoption, especially in such environments - like HPC - where Windows has not been popular.

Windows Server 2008 introduces some significant improvements that address most of those issues:

1. Configuration validation
A test tool is built into the product. It will analyze nodes and shared storage (if any) before they join a cluster. It can also be used as a troubleshooting tool, as long as the storage you want to analyze is offline. The tool will point out any issues with the hardware and the configuration that may make them unsuitable for a fail-over cluster. It will finally replace the cluster HCL. So, if the hardware passes validation, then the configuration is officially supported.

2. Simplified resource setup
A wizard-driven process allows you to select which roles you want to cluster (e.g. file server, print server, virtual server), then sets up cluster resources and dependencies appropriately.

3. Improved SAN support
Windows server 2008 issues persistent reservations on shared storage to establish ownership of LUNs, it does not use bus resets any longer. Bus resets are disruptive on SANs where several systems on several platforms may share the same storage bus. This implies that the storage must support persistent reservations. Shared parallel SCSI is deprecated.

4. Changed quorum model
The administrator can choose the most appropriate quorum model for the configuration. Several are possible:

Majority node sets with witness disk: each node gets a vote and so does a witness disk. The cluster will survive the failure of any 1 vote (including the shared witness disk).
Majority of nodes: storage is replicated amongst them, but does not get a vote. The cluster is active until the majority of nodes is running.
Majority of nodes and file share witness: nodes get a vote and so does a file share on a separate server. This is ideal for geo-clusters, as the witness file share can be in a 3rd site. The geo-cluster will thus survive the loss of any one site.

5. Improved networking
The nodes need no longer be on the same private subnet and the timeout of the “ping” among nodes is configurable. This makes it possible to route private traffic between locations and removes any a-priori restrictions on distance. Obviously, practical restrictions remain and will depend on how much the clustered applications and their users will tolerate.

These are just a few of the innovations available in server 2008 clusters. You may want to try them out for yourself by building a simple cluster on a set of virtual machines. You don’t need shared storage any longer, but if you want to try a quorum with witness disk, you can set up one of those machines as an iscsi target.

Get started

If you want to try Windows Server 2008 clusters, a virtual lab is available on Technet Events.
There is also an excellent screencast by David Northey on http://edge.technet.com/Media/Windows-Server-2008-Failover-Clustering/
Official training will be launched shortly.

HPC Server 2008 webcasts

gmarchetti — Wed, 23 Jan 2008 03:07:00 GMT

My introductory demo / webcast on HPC Server 2008 beta 1 has been recorded. It can now be viewed here.

I have received feedback from several people about it, requesting instructions to build the virtual machines I used and to repeat the demos. Hence, I have written a short document that will guide you through the set-up of a virtual HPC Server 2008 cluster. It also provides you with some examples of what is possible with the new administration, diagnostic and scheduling tools. It is intended as a hands-on introduction to the salient features of HPC Server 2008 beta 1. It is not an official click-by-click manual and it must be considered work in progress. I will update it when beta 2 is released to cover new or changed features. No warranty is implied nor support promised. You try at your own risk ☺

The document is now available on https://windowshpc.net/Resources/Documents/2008b1_labs_and_demos.zip

Windows HPC Server 2008

gmarchetti — Wed, 14 Nov 2007 11:02:09 GMT

The software previously known as Compute Cluster Server v2 is now available in beta 1 on http://connect.microsoft.com

HPC Server 2008 contains some pretty significant innovations. I'll summarize them here and then produce some more detailed entries on each of them. Also check out the videos about v2 on http://edge.technet.com, (mostly) courtesy of yours truly :-)

Noteworthy innovations in v2 for IT Pros:

1. It runs on Longhorn Server (oops, Windows Server 2008) only. There is no upgrade process. Wipe and replace.

2. It uses Windows Deployment Service (WDS), not RIS, which makes that "wipe and replace" much less painful :-) Multicast deployment is supported :-)

3. The administration console offers a one-stop shop for deployment, administration, diagnostics and reporting. I know that many of you will be thinking, "just like Ganglia!". Well, that's the general idea. It is not full system center, but it is a very functional and efficient way to manage a HPC cluster.

4. It will offer head node fail-over from beta 2 onwards, thus eliminating a worrying single point of failure. This feature uses Server 2008 fail-over clustering, so it requires enterprise edition or better for the head node.

5. As a consequence of (4), we will support installing the head node on a sql 2005 cluster. In fact, we include sql 2005 express with the product but also support installations on pre-existing sql 2005 servers. You need not install the head node services on the sql machine either.

6. We have devised a new networking api to run along winsock direct, called Network Direct. the idea is to enable verb-based interaction with low-latency networking hardware, thus shaving off another couple of microseconds of latency, much like it happens with MVAPICH. In this release the only consumer of network direct is MSMPI. We're working with OEMs to write network direct providers / drivers.

7. Powershell scripting is used for administration of common operations, along with the old v1 commands. In fact, those still work perfectly because we have maintained 100% compatibility with v1 COM API. Of course the v2 API exposes new functionality, but that deserves a post in itself.

8. The scheduler has been significantly enhanced for scalability and optimization. It deserves a post in itself, but here are some significant changes:

- ability to dynamically grow and shrink the pool of resources allocated to running jobs

- enforcing constraints on the basis of job templates, not just filters

- use of different units of allocation: core, CPU slot, node, depending on what your application needs

- biasing allocation algorithm towards memory or CPUs

- from beta 2 onwards, pre-emption of running tasks !!

9. Last but not least by any means, we are working with partners to support clustered file systems. CXFS, Melio and StorNext FS come to mind, being available on Windows now (2008 support is in the works). More are coming.

Of course this is a beta product and the usual caveat applies: features you see may not make it into the final product, may not be fully functional, etc... Still v2 is definitely worth a try, because of the great improvements.

Note that I have spoken just of those topics of interest to IT Pros. There is more in the works for developers, like the ability to interface with the cluster using WCF and schedule WCF services.

By the way, we are running a program called HPCPAL for those of you interested in trying it out. We offer help with design of either infrastructure or software architecture followed by 4 days on site in Redmond with us, working on your code. All we ask for is a reference. If you're interested, send a note to hpcpal@microsoft.com and we'll take it from there.

I hope I've not forgotten anything important... I'm jet-lagged in Barcelona, where I'm speaking at Teched IT Forum. If you're around and want to chat about CCS, drop me a note.

In the meanwhile, check out our website, where you'll also find whitepapers describing hpc server 2008 in more detail.