HPC, Virtualization and Random Thoughts : HPC

Updated Infiniband on Server 2008 Paper

gmarchetti — Tue, 03 Mar 2009 04:31:21 GMT

I have finally updated my notes on the installation of Infiniband on Windows Server 2008. They now cover the released version 2.0 of Mellanox WinOF stack. You can find the document in my skydrive public folder.

Let me know if you find it useful.

Powered by Qumana

Faking Networks

gmarchetti — Tue, 24 Feb 2009 01:46:26 GMT

On a Windows HPC Server 2008 head node, that is...

1. No Infiniband on the head node

In many cases people want to save themselves some money by not installing an Infiniband adapter on the head node, thereby also sparing a port on that expensive infiniband switch. It makes a lot of sense, especially when you plan not to perform any calculations on such machine. So, how do we make the software believe it has an Infiniband adapter?

The HPC management tools do not care too much about the type of connection you have, as long as they can get an IP address to communicate with. So, you can install a "loopback adapter", give it a fixed IP address and pretend it is a real network card. Of course, you will not be able to use it to communicate with the compute nodes, but if all you want to carry on IB is MPI traffic amongst those, the trick will work.

The only caveat is that you lose the ability to use dhcp on the infiniband network, hence you will have to provide a mechanism to assign fixed IP addresses for IPoIB communication. Of course the subnet you use on the "fake" IB and the real one must be the same.

The easiest way is possibly to write a small script that uses the netsh command, then run it on all the compute nodes. You will need at least 1 private Ethernet network for management traffic across the cluster.

For instance, the command below will assign the ip address 192.168.3.100 and a 24-bit mask to the network connection called "Application"

netsh int ip set address "Application" static 192.168.3.100 255.255.255.0

2. No public ethernet

In several cases I found that the head node has only 1 ethernet card. Our HPC software out of the box prevents the use of Windows Deployment Services, DHCP unless you have at least 2 adapters, in order to avoid conflicts with existing deployment solutions. You may choose to install a fake "public" network on a loopback adapter and thus enable WDS on the real "private" network.

3. No private ethernet

Another interesting case is the one you get with many pre-built clusters, which provide 1 Ethernet and 1 Infiniband network in the box.

Note that when you install an Infiniband stack (e.g. WinOF 2.0), you typically get an IP-over-IB protocol provider. Thus, it is possible to use the infiniband network to route private cluster traffic, with the exception of deployment (no PXE-boot over IB). For "heavy" mpi applications, you will want to keep the two networks separate anyway.

Powered by Qumana

Upgrading from an evaluation version

gmarchetti — Tue, 04 Nov 2008 02:39:00 GMT

I have received a few questions about upgrades from the evaluation version that you can download from microsoft.com/hpc to a full version.

The good news is that the evaluation version is fully functional, so you won't need a complete re-installation. The only thing you need to do is obtain a full licence key, then:

- To upgrade the hpc pack tools you have to run “upgrade.exe” on the head node. The hpc pack CD contains the upgrade.exe file.

- To upgrade the o/s, you have to obtain a full licence key for all the nodes, then run slmgr.vbs –ipk <new licence> across the cluster. You can do that from the command line (clusrun /all) or via the GUI.

You can also use slmgr.vbs to extend the evaluation period by another 60 days. When you are approaching the end of the evaluation, simply run slmgr.vbs -rearm across the cluster. Note that the evaluation does not require activation, but a full licence does.

Please see http://support.microsoft.com/kb/948472 for more information.

Proxies and Compute Nodes

gmarchetti — Thu, 23 Oct 2008 00:22:40 GMT

You’ve prepared your templates, configured your network, your firewalls and everything you could think of, yet your automated provisioning takes forever and eventually fails…

Well, check if you have a patching task in your node template. If you do, you’ll need a way to reach the Microsoft Update service and download any patches. You may need to set a proxy on the nodes for that. Alas, the GUI interface does not offer you an option to do that. Also, any proxy setting that you specify in Internet Options is effective just for the logged-in user. So, how can you set a proxy for windows update to use?

The Windows Update service uses the WinHTTP protocol. You can set a protocol-level proxy with:

netsh winhttp set proxy proxy-server=”http=<your proxy:port>” bypass-list=”<local>”

Where <local> is typed literally <local>. You could have that command line run before the patching task in the template.

Alternatively, you could deploy the nodes without the patching task, run that command across the cluster, then apply a template with a patching task.

Last but not least, you could set up a Windows Update Server on your corporate network and then use group policies to direct the update service on the nodes to that server.

Anyway, if your nodes go anywhere near the Internet, please keep them patched!

Upgrading to HPC Server 2008 RC1

gmarchetti — Wed, 23 Jul 2008 01:37:00 GMT

Well, there is no upgrade path, so the quickest way is to re-image.

Download the RC build of the HPC software from connect.microsoft.com. If you have Infiniband cards, download the latest WinIB-ND drivers (1.4.0.2577) from http://www.mellanox.com.

1. Re-image the head node and install the latest HPC pack.

2. Unzip the WinIB package on the head node, e.g. to c:\ib.

3. Open device manager and update the drivers for the Infiniband adapter, then the openib adapter. 4. Point the wizard to c:\ib\inf to select the appropriate IB driver. Add any other drivers as required.

5. Build a new o/s image (or re-use a previously built one)

6. Click on "manage drivers" in the to-do list. Point the wizard to c:\ib\inf to add the drivers to the image

7. Create a new node template with the image you built

8. Reboot the compute nodes and wait for re-deployment to complete

9. Use clusrun to copy c:\ib from the head node to the compute nodes

10. Use clusrun to run c:\ib\inf\ndinstall -i on all nodes and thus install the new ND provider.

If you are using OpenSM, you'll find a new version of it in c:\ib\tools.

You may also have to re-boot the Infiniband switch after the driver update. I haven't figured out why yet, but IPoIB worked without problems, MPI over ND did not just after the driver update. Rebooting the switch seemed to fix this.

I'll be there in 2 microseconds!

gmarchetti — Thu, 19 Jun 2008 20:00:38 GMT

Fantastic news! Mellanox has released the beta 2 version of their WinIB 1.4 stack, which works with HPC Server 2008 beta 2 and has Network Direct providers for their latest ConnectX cards. The results announced at ISC 08 are outstanding:

- 2 microseconds' latency

- 2 GB/s throughput

Another outstanding result for HPC Server 2008 is the Umea cluster, at n. 39 in the Top 500 list:

- 46.04 TFlops

- 85.6% efficiency

Hats off to the Network Direct team. Now Windows HPC Server 2008 plays with the big boys :-)

Powered by Qumana

Teched session on HPC in top 20!

gmarchetti — Wed, 18 Jun 2008 19:14:24 GMT

Phil Pennington and I presented a session on cluster performance optimization at Teched 2008 in Orlando. It made it in the top 20 list by customer satisfaction!!!

To all those who were there and voted for us: Thank you!!!

To all those who were not there but would still like to know about it: leave a comment!

Powered by Qumana

How fast is this thing, really?!

gmarchetti — Wed, 18 Jun 2008 18:33:55 GMT

Microsoft entered the HPC market a couple of years ago with a value proposition based on ease of integration, use and management. The comment we received most often sounded more or less like this: "This is all well and good, but how fast is this thing, really?". Well, here are a couple of impressive answers:

1. The NCSA cluster, running Windows HPC Server 2008 CTP on 1184 nodes (9472 cores), achieved 64.48 TFlops and 77.7% efficiency. This places it at n. 23 of the June 08 Top500 list.

2. The Aachen cluster, running the same build on 262 nodes, achieved 18.81 TFlops and 76.5% efficiency, which places it at n. 100 of that list.

Happy now? ;-) If you want to read more about the details, have a look at http://www.microsoft.com/hpc

Powered by Qumana

A Hybrid OS Cluster Solution

gmarchetti — Tue, 03 Jun 2008 22:28:09 GMT

Thomas Varlet, of Microsoft France, and Dr. Patrice Calegari, of BULL SAS, have written an excellent paper on how to build hybrid clusters, i.e. clusters where 2 or more operating systems can be run at the same time. It is recommended reading, in my opinion, for those of us who use both Linux and Windows HPC solutions. You'll find the paper here.

UK HPC User Group Meeting

gmarchetti — Tue, 03 Jun 2008 03:42:25 GMT

The UK HPC user group is meeting in London on June 26th, for what promises to be an interesting day at the Imperial War Museum.

This meeting is intended for customers, partners and developers to “meet and mingle”, compare notes and provide Microsoft with direct input into our product and offerings.
On the day the attendees will hear:
-         The latest news on Windows HPC 2008
-         New software solutions in Finance, Engineering and Defence
-         The latest in MS enabling technology like Microsoft ESP & MS Robotics
-         The winners of the UK HPC student competition
-         Customer stories.
Attendees will also have the opportunity to tour the Imperial War Museum.

Please register here.

The event is being organised by the UK Microsoft HPC User Group, chaired by Professor Simon Cox, School of Engineering Sciences of the University of Southampton.

Powered by Qumana

Webcasts Again!

gmarchetti — Fri, 23 May 2008 19:32:00 GMT

Hey all,

I'm doing another series of webcasts. 2 of them have already been aired, two more will happen shortly. Here are the topics:

- deployment and management

- high availability

- new scheduler features

- hpc server 2008 and linux

You'll find a link to register and summaries on:

http://www.microsoft.com/hpc/events.aspx

Affinity

gmarchetti — Tue, 13 May 2008 02:42:53 GMT

Did you notice that the latest CTP has introduced a new option for mpiexec? Using mpiexec -affinity you can affinitize the mpi rank to the core where it is started, thus avoiding context switches. Your application will determine whether you actually benefit from affinitization or not. Some of them show a good performance improvement, some do not. In particular, if you have an MPI application that is also multi-threaded, the affinity option may backfire, because the affinity mask that you set for the process is inherited by default by all its threads. Thus, its threads may be stuck on 1 core. Windows offers other API calls to set thread affinity.

"Traditional", non multi-threaded MPI applications may be more straightforward. One important factor to take into account when deciding when to affinitize the process is the compute node architecture: is it NUMA or not? If it is, have you got enough RAM in the memory bank local to the core where the process will run? If not, you may incur frequent (and lengthy) remote memory accesses on the same hardware. In this case, it may be best to rely on the o/s scheduler to determine the ideal NUMA node for the thread.

Powered by Qumana

4x4 and other quick tips

gmarchetti — Thu, 08 May 2008 02:21:28 GMT

I've recently been involved in a simple benchmarking exercise. Here are a few quick "rules of thumb" that have helped me:

- 4x4

A PCIe 4x slot is supposed to have 4 lanes capable of 250 MB/s each, for a total of 1 GB/s. An Infiniband SDR 4x card has 4 channels clocked at 2.5Gb/s, so a simple rule of thumb is: put an Infiniband card in the PCIe slot with the same number of channels. This is not a coincidence: Intel was part of the original Infiniband group.

BUT

Be aware that not all motherboards are equal, although in theory most of them use the same chipsets. In our case, we found out that the motherboard was not able to sustain more than about 600 MB/s on the PCIe 4x slot. We had to move the Infiniband cards to the 8x slots, where we could reach the expected 900 MB/s transfer rate of the card. The 8x slot on those motherboards is probably not capable of reaching its top speed either, but it is sufficient for the SDR 4x card.

- Snoop Filters

A snoop filter is a mechanism to reduce traffic between different memory bus segments. It is particularly useful in multi-cpu, multi-core machines. Applications generally benefit from it, but there are some cases where latency-bound applications are adversely affected. If you see erratic behaviours in your latency tests (e.g. "random" high latencies in an otherwise consistent benchmark) and you have quad-core machines (especially early Clovertowns), try and disable the snoop filter in the bios. It may (or may not) help. Again, motherboards affect the results, as different components (with or without snoop filter) were used by different manufacturers.

New quad-core machines (Harpertown) have a snoop filter, but do not seem to show the symptoms mentioned above (at least those I've seen).

- Dynamic Power Management

It is generally NOT a good idea when you're trying to squeeze the last FLOP out of the CPUs. Disable it in the BIOS.

- MPI traffic

You may want to make absolutely sure that your MPI applications are using Infiniband; or you may want to run them once on Ethernet and another time on Infiniband, then compare the results. In any case, you can specify the network where MPI traffic will go at run time:

mpiexec -env MPICH_NETMASK <address>/<mask> <other parameters> <exe>

You may also want to make absolutely sure that your MPI traffic uses Network Direct, not winsock. You can:

- remove the Winsock provider. Coarse, but effective:

clusrun /<nodes> installsp -r

- run your application with

mpiexec -env MPICH_DISABLE_SOCK 1 <other parameters> <exe>

Incidentally, you can install the Network direct provider with

clusrun /<nodes> ndinstall -i

HPC & Movies

gmarchetti — Fri, 02 May 2008 01:51:00 GMT

In my last post I investigated how HPC can be used to build UFOs. This time, I've learned to my surprise that HPC can be used to make movies!

Digital media production follows a complex workflow, from initial sketch to wireframe model, to rendered 3D images, to movies. HPC is typically used in rendering, encoding or transcoding. I've done some research on the matter and posted the results here.

Let me know what you think of it.

Interested in HPC? How about UFOs?

gmarchetti — Fri, 21 Mar 2008 22:43:00 GMT

To many people HPC is like UFOs: We there's somebody somewhere, but we don't really know what they're doing and where they fit in the grand scheme of things.

Here's my attempt at explaining them (UFOs AND HPC). Happy reading.

If you think I've smoked one too many - please leave a comment. Equally, please let me know if the article makes sense to you.