The Windows HPC Team Blog : Deployment

How to set environment variables on computes nodes

Thomas.varlet@MS — Tue, 07 Apr 2009 21:59:00 GMT

For ISV application you have to set some environments variables on the compute node.
You can set this variable by hand, but if you have many nodes it can be long.
So, the idea of this post is how can set them attaumaticaly ?

Use the HPCS tools :

You can set cluster wilde variable by using cluscgf in a Administrator cmd line windows :
"cluscfg setenvs name=value"

ex. : cluscfg setenvs LM_LICENSE_FILE=license_server@74000

So each time you will submit a job this variable will be set for the job

Use command line :

Select the nodes into the node management view and run the following command
"setx name value /M"
This command will add this environement variable in the system of all the selected nodes

NOTE : /M set the variable for the system

ex. : setx PATH "%PATH%;\\headnode\Apps\bin" /M

Helpful tip for dealing with HPC Pack setup failures

ihimmar — Fri, 03 Apr 2009 12:20:00 GMT

I once ran into an issue while installing the HPC Pack on a machine and I was wondering where to go to find more information about the setup failure preferably a log of some sort to see the reason for the failure. Here is a tip that helped.

The HPC Pack setup writes two logs to the installer’s %temp%\HPCSetupLogs directory: one for the MSI related installs and one for the setup program itself. If you run into errors while installing, more information would be present in setup-<date>-<time>.txt log.

Simply navigate to your %temp% folder and look for the HPCSetupLogs directory and you can see a logged output of the setup.

Hope someone finds this helpful someday.

HPCv2RC1: Nodes imaged successfully, but show “Provisioning Failed”

DougLi — Mon, 30 Jun 2008 14:05:00 GMT

I just installed the brand new RC1 candidate on my head node. I provisioned all 23 nodes of my cluster at once, using the Default compute node template. While going through the “Create a node template” wizard in the To-Do list, I selected the option “Include a step in the template to download and install updates for my cluster using Microsoft Update or the Enterprise Windows Server Update Services (WSUS)."

I monitored the progress of the nodes by looking at the “Provisioning Log” tab of each node in the “Node Management” view of the HPC Cluster Manager console. Everything went smoothly through the operating system installation phase. All nodes downloaded and installed an operating system image. To my dismay, however, every node then failed the provisioning process at the same step: “Windows update failed to find updates. Exception from HRESULT: 0x8024402C -2145107924".

For a host of reasons, not the least of which are external connectivity issues or misconfiguration of the head node, compute nodes may be unable to locate the WSUS server on first try, and will retry. However, after 3 failures to locate WSUS, the overall provisioning task will be deemed a failure. In this case, I was left with all nodes in the “Unknown” state with Node Health set to “Provisioning Failed.”

At this stage, I knew of two obvious options:

1.) I could diagnose and fix whatever connectivity issues were affecting WSUS and retry the existing provisioning process

2.) I could modify the default compute node template to eliminate the WSUS task, reprovision all nodes, and then once the provisioning completed successfully apply a second template that just handled WSUS.

However, there is a third option that eliminates the need to do a full reprovisioning. With this option, I was able to salvage the operating system installation on each node and bring all of them into a provisioned state. That option? Delete the compute node from the HPC Cluster Manager.

When you delete a compute node that otherwise has a healthy installation of the HPC node manager and related services, the compute node will attempt to re-connect to the head node. This process may take several minutes, but eventually the node will show up once again in the HPC Cluster Manager console, with a status of “Unknown” and a “Node Health” of "OK". At this point the two options become:

1.) Diagnose and fix connectivity issues, re-apply the same provisioning template. This time, the node manager will be smart-enough to recognize that the node is already installed, and will skip over the installation steps – saving time and bandwidth.

2.) Modify the existing template to remove the WSUS step and/or create and apply a non-imaging node template.

I chose the second option and successfully provisioned my nodes. Next step: chase down the source of my connectivity issues…

Virtual HPC Cluster Deployment on Hyper-V

Yutong Sun — Sat, 26 Apr 2008 02:31:00 GMT

If you want to setup and deploy a cluster with Windows HPC Server 2008 April CTP build on a single laptop or desktop, or if you want to build a bigger cluster with more nodes on a few powerful machines, here is a step-by-step document to help you achieve this by using Windows Server 2008 Hyper-V. (See the attachment.)

How to use C shell and Korn shell to manage a Windows HPC Server cluster?

Zhen Wei — Fri, 18 Apr 2008 02:43:00 GMT

(On behalf of Christina Carter)

Did you know you can use the familiar C shell and Korn shell under SUA (Subsystem for UNIX-based Applications) to call our PowerShell commands? If you have Windows Server 2003 R2, Windows Server 2008 or Windows Vista Ultimate and Enterprise editions, then you can install SUA and try it out.

Here are the steps:

1. Install SUA via Add/Remove Windows Components. See this post for how to do that on Vista, and see this one for Windows Server 2003 R2.

2. Here are a few commands you can use in C Shell (You need to substitute "headnode" with your head node name)

cd /dev/fs/C/Windows/System32/WindowsPowerShell/v1.0/
./powershell.exe -command get-hpcnode -scheduler headnode
alias nodelist ./powershell.exe -command get-hpcnode -scheduler headnode
nodelist
nodelist | grep Online

See this post on more info about how PowerShell and SUA work together. There's a comment in this blog also talks about using class UNIX commands in SUA. (Search for the "SUA" wording)

Hardware Configuration for Larger Clusters

DougLi — Fri, 16 Nov 2007 17:00:00 GMT

Hi, I’m Doug and I design & run production Windows Compute Cluster Server clusters for the HPC team at Microsoft. I often get asked “what hardware configuration should I choose for my cluster?” The stock answer is: “it depends.” WCCS will run just fine on two desktop machines connected by an Ethernet mini-hub, but that may not be sufficient for your application’s performance needs. For customers with demanding applications and larger budgets, a hardware purchasing decision can be quite complex.

The HPC Test team bought a 256-node SDR-Infiniband-based cluster, named Rainier, one year ago. The Rainier cluster achieved a ranking of #116 on the November 2007 Top500 list at www.top500.org, after having been in operation for less than 2 weeks. Rainier has 256 compute nodes and is primarily used for testing new builds of the product as we work on the next version of Windows Computer Cluster Server. People are curious about the hardware configuration of the Rainier cluster and the decisions that go into making a sizeable cluster purchase.

At today's prices, a 256+ node cluster with high-speed network interconnects is likely to cost in excess $1M when all hardware, cabling, facilities modifications, and installation labor is factored into the total cost.

The three primary and most difficult challenges in planning hardware for a large HPC cluster are:

1.) Where will you put it?

2.) How will you power it?

3.) How will you cool it?

Answering these three questions is a significant challenge for those of us who build HPC clusters in lab environments here at Microsoft and for customers I have talked to. Many modern datacenters are not built to the power and cooling specifications necessary to achieve the kinds of hardware densities that are available with today's server hardware. A per-rack power-consumption spec for the average datacenter constructed in the past 10 years is typically ~2-6kW. If you are building a new datacenter to support HPC workloads, a design target for power consumption per-42-U-rack of ~15 kW will support many of the higher-end blade-based configurations for some time to come.

How well you can power and cool the server and network hardware helps greatly to determine

a.) How densely you can populate the server racks and thus

b.) Which server form-factor is an option for deployment and thus

c.) What the most cost-effective physical network design will be

Power and cooling will dictate whether or not it is feasible to fully populate a 42-U rack with blade-based or traditional 1-U-server-based form factors.

Server blades typically offer

a.) the greatest density of CPU cores per-U

b.) streamlined management and monitoring (no need for KVM switches & KVM cables, for example)

c.) ease of replace-ability for individual components or servers

d.) Power consumption of 2-4 kW per fully-populated blade chassis.

1-U form-factor systems typically offer:

a.) lesser CPU core density per-U than blades

b.) more overhead in managing out-of-band configuration

c.) better cable management solutions

d.) Greater total-memory and local disk expansion per-server

e.) Greater expansion options (co-processors/GPU's, etc.)

With the ever-increasing server densities in hardware racks, weight is a fourth consideration that cannot be ignored. In some of our older lab facilities (housed on upper floors of standard office buildings), we are prevented from fully populating 42-U racks with blades or 1-U servers due to weight restrictions.

The HPC Test team settled on a Dell PowerEdge 1955 blade configuration for the Rainier cluster (See http://download.microsoft.com/download/4/7/8/478f369c-f530-4a1f-a9d8-2d219d42c297/Windows%20HPC%20Server%202008%20Top500%20Datasheet_11-07.pdf for details). The choice was driven by pricing and the availability of the then-new quad-core Xeon processors. We manage two other smaller “production” clusters: an HP cluster based on the 1-U DL145G2 platform, and an IBM 1350 cluster (purchased as a Linux cluster and converted to Windows two years ago). All 3 hardware vendors have blade and 1-U server offerings. Every vendor approaches system management slightly differently, so you will want to evaluate their management frameworks in the broader context of how they will fit within your existing server management infrastructure (for a complete list of Microsoft HPC partners visit: http://www.microsoft.com/windowsserver2003/ccs/partners/partnerlist.mspx).

Having an understanding of applications requirements ahead of making a server platform decision is useful because it will prevent over- or under-sizing choice of CPU and memory. Unfortunately, when you do not know what the applications requirements will be for the cluster, or when a cluster will be used for many different current-and-future applications, planning an optimal hardware configuration is a lot harder. For a general-purpose cluster supporting multiple applications, a baseline rule of thumb is to have no less than 2GB of RAM per-CPU-core. So for a dual-processor/quad-core compute node, you should expect to have a minimum of 16GB of RAM.

One of the problems we ran into with deployment of server blades in our lab datacenter facility was insufficient cooling. Without adequate cooling, the facilities owners were not willing to let our team fully populate a 42-U rack with blade chassis. Instead, we were allowed only to populate to a maximum of 80% of rack capacity. If we had opted for 1-U systems, we would likely have realized between 36 and 42 systems per rack. In our case, then, the total number of processors per rack was roughly the same regardless of platform choice.

One side-effect of spreading the servers out into a greater number of less-densely-populated racks was that it complicated the network design. Fewer racks == shorter cable runs to each rack from a network switching equipment location. Copper cables have stringent distance limitations. Different server platforms and network adapters have a different maximum-length of cable that they will support. In the case of the Dell PowerEdge 1955 blades, each blade supported a maximum 5M cable length for copper Infiniband cables. I have two recommendations based on this experience:

1.) If at all possible, avoid the use of copper cable for high-speed networks in large clusters. Copper is fine for a standard Ethernet management/monitoring network within the cluster. But for dedicated high-speed application networks, fiber-optic cabling is easier to work with, label, identify, replace, etc. I was surprised just how heavy and unwieldy Infiniband cables become when there are in excess of 500 in a confined space. Fiber optic cabling can be expensive, but Infiniband cables are already expensive; the cost premium of fiber is worth it. At least one vendor now offers a fiber-optic cable with standard connectors for existing copper switches and host adapters.

2.) If possible, streamline the network design by using larger centralized switches. It is easier to centralize the switch design with a small number of densely-populated racks. Fewer switches means fewer possible points of failure and less hardware overall to manage. The Rainier cluster design, unfortunately, consists of a dedicated Infiniband switch per blade chassis. This design was a necessary compromise due to cost and cable length restrictions. One note of caution: avoid lower-cost technologies which make use of oversubscription at the switch level and (in the case of blades) at the port-concentrator level. A cluster that routinely has compute nodes competing for the same individual bandwidth is a cluster that will not have fully-utilized CPUs.

There are multiple choices for choice of high-speed network technology. Traditionally, the two choices in HPC have been Infiniband or Myrinet. We run production clusters here at Microsoft based on both technologies. Each has its' own advantages and specific configuration settings, but they are both relatively easy to support on the Windows platform. It should be noted that major network vendors are also offering 10Gb-E solutions now, and network switch vendors are making great improvements in switch density to help drive down costs. My recommendation regardless of technology is to choose the vendor that you have the best relationship with, who provides the best hardware and driver support for your specific application and system needs, and who will work with you to spec your final design prior to purchase.

Hope this helps.

-Doug Lindsey

(dougli@microsoft.com)

Microsoft HPC

Note to the following Deployment posts

MichaelCole — Tue, 16 Oct 2007 16:56:00 GMT

The following posts are exact copies of entries in the same titled blog on the previous Windowshpc portal site. I added the author's name in the beginning of each entry and edited the posting dates to reflect the original date of the entry.

Michael Cole
Project Manager/Site Administrator

University of Southampton provides Windows access to the UK National Grid Service

MichaelCole — Thu, 20 Sep 2007 18:10:00 GMT

posted Thursday, September 20, 2007 4:11 PM by DennisCr |

The University of Southampton is the latest university to join the UK National Grid Service with affiliate status. The Microsoft Institute for High Performance Computing at the University will be providing access to the Microsoft Compute Cluster Service through a Globus gateway, the first instance of a Windows cluster being incorporated into a production level Grid. Prof. Simon Cox, Director of the Institute said “This is an exciting opportunity to offer, for the first time, Microsoft Windows Compute Cluster Server on the National Grid Service, where it will interoperate with other resources”.

A large proportion of the UK research community primarily use Windows based systems or are more familiar with Windows than other operating systems. The implementation of the Globus Gateway will allow the NGS to be accessed by a much larger number of researchers who will be able to carry out research under a more familiar operating system.

The CCS Globus Gateway provides access to a cluster under the Windows x64 platform using the normal Globus client tools and Grid security mechanism, therefore makes it possible to share the Windows based computational resources with a wider user base in the e-Science community.

About the University of Southampton

o The Microsoft Institute for High Performance Computing at the University of Southampton is part of the University's School of Engineering Sciences, which offers expertise in engineering disciplines that underpin every aspect of 21st century life. Evolving from its traditional engineering roots, the School continues to expand into new, exciting research areas. From orthopaedic implant modelling and modern fuel cell technologies, to enhancing satellite broadcast technology and improving the aerodynamics of Formula One racing cars and hydrodynamics of America's Cup yachts, the School is always at the leading edge of engineering research. www.mihpc.net

o The University of Southampton is a leading UK teaching and research institution with a global reputation for leading-edge research and scholarship. It is one of the UK's top 10 research universities, offering first-rate opportunities and facilities for study and research across a wide range of subjects in humanities, health, science and engineering. The University has around 20,000 students and over 5000 staff. Its annual turnover is in the region of £310 million. www.southampton.ac.uk

9,200 Core Windows CCS Cluster

MichaelCole — Thu, 30 Aug 2007 17:40:00 GMT

posted Thursday, August 30, 2007 3:39 AM by saptak | 0 Comments

Actually, the cluster is a dual boot Linux/CCS cluster, which makes it even more interesting than the headline. The machine is reported to have 1,151 Dell servers with dual-socket Barcelonas for a total of over 9,200 cores and will be housed at the University of Nebraska, Omaha.

We recently published a whitepaper which details howto setup such Linux/CCS dual boot cluster. Go grab it @ http://www.microsoft.com/downloads/details.aspx?FamilyID=1457bc0a-eaff-4303-99ed-b199ab1c0857&DisplayLang=en

Call for comments for Microsoft Operations Manager Management Pack

MichaelCole — Mon, 22 Jan 2007 15:30:00 GMT

posted Monday, January 22, 2007 12:29 PM by chris | 0 Comments

We released the MOM Management Pack for Windows CCS back in November last year. You can download the pack at http://www.microsoft.com/downloads/details.aspx?FamilyId=53AE834A-7369-4655-82E5-1C2563CEC677&displaylang=en. You can also download the pack guide at http://www.microsoft.com/mom/techinfo/productdoc/default.mspx#E2BAC.

Anybody has been using the pack? Do you have any suggestions, comments, etc? We would love to hear from you and do the best we can to update the pack to suit your needs.

SGI and Microsoft Join Forces to Expand High-Performance Computing

MichaelCole — Mon, 15 Jan 2007 12:10:00 GMT

posted Monday, January 15, 2007 9:11 PM by DennisCr | 0 Comments

Leveraging SGI’s expertise in high-performance computing and Microsoft Corp.’s strategy of making HPC more accessible to a broader marketplace, SGI and Microsoft announced that SGI will offer Microsoft^® Windows^® Compute Cluster Server 2003 on SGI^® Altix^® XE cluster systems based on quad-core and dual-core Intel^® Xeon^® processors. SGI Altix XE cluster systems with Windows Compute Cluster Server will be available beginning in March 2007.

more...