The Windows HPC Team Blog : Applications

Process Affinity and Windows HPC Server 2008 SP1

Barndawgie — Thu, 01 Oct 2009 23:57:00 GMT

An issue came to our attention after Windows HPC Server 2008 shipped regarding the way we set affinity on the processes within a job. There are actually two places where we set affinity:

1. The Node Manager (the service on each node responsible for starting jobs and tasks) sets processor affinity on each task to prevent that task from using processors which are not assigned to it.

2. MPIEXEC (which is used to start MS-MPI applications) can, when the –affinity flag is provided, set affinity on all ranks within the MPI application.

The problem that we encountered is this: Due to the way affinity setting works on Windows Job Objects (which we use to run tasks) and the processes within them, you cannot set affinity at both layers. That means that in MPI Tasks which are allocated less than an entire node, the –affinity flag will end up being ignored on the MPIExec command line, since the affinity has already been set by the scheduler and cannot be set in two places. This caused problems for some applications, especially those developed to work against the Compute Cluster Pack (which didn’t set affinity at all).

The problem is particular serious for jobs which specify the –exclusive option; when a job specifies the –Exclusive option it will be allocated an entire node. But the scheduler will set affinity on tasks within the job despite this. So an exclusive job with a 4 core task that is assigned an 8 core node would cause the scheduler to affinitize the task to only 4 cores: This leaves the other 4 cores idle if there are no other tasks in the job and is awfully confusing for some people and applications! Such a job would also not have MPI rank affinity, even if the –Affinity flag was specified.

Our solution is to introduce a new cluster parameter called AffinityType. AffinityType has three possible settings which work as follows:

· AllJobs- When AffinityType is set to AllJobs, the Node Manager will set affinity on any task that isn’t allocated an entire node. This is the behavior described above, and is probably the best choice for applications which may run multiple instances per node (e.g. Parameter Sweeps and SOA Jobs) and want these instances to be isolated from each other.

· NonExclusiveJobs (Default)- With this setting, the Node Manager will not set affinity on jobs which are marked as exclusive. This is the ideal choice for jobs with only 1 task, since that task will be able to take advantage of all cores on the nodes that it is assigned. We’ve made this the new default since it provides what is generally the preferred behavior for MPI tasks, which are most likely to be sensitive to affinitization. With this choice selected, MPI tasks in exclusive jobs can take advantage of the –Affinity flag to MPI even if they are not allocated an entire node.

· NoJobs- With this setting, the Node Manager will never set affinity on any task. This is an excellent choice for those running MPI Tasks who want to make sure they can take advantage of MPI’s –Affinity flag even when jobs may share nodes. This is also useful for applications which want to set their own affinity.

Note that Windows Server 2008 R2 will allow the setting of process affinity at both the Windows Job Object and Process level simultaneously. So hopefully in v3 of the HPC Pack there will no longer be an issues with the conflict between these two setting.

You can learn more about how the new AffinityMode flag works here: http://msdn.microsoft.com/en-us/library/microsoft.hpc.scheduler.properties.affinitymode(vs.85).aspx

Weather Research and Forecast (WRF) Model Port to Windows

MichaelCole — Thu, 30 Oct 2008 13:04:00 GMT

The Weather Research and Forecast (WRF) project is a multi-year/multi-institution collaboration to develop a next generation regional forecast model and data assimilation system for operational numerical weather prediction (NWP) and atmospheric research. Under this project – a collaboration between The National Center for Atmospheric Research (NCAR), Microsoft Corporation, Advanced Micro Devices, Inc., and The Portland Group, Inc (PGI) – a prototype version of WRF has been developed and demonstrated running in parallel using MPI on an AMD Opteron™ dual-core processor based server running Windows Compute Cluster Server 2003, Microsoft Compute Cluster Pack,. This preliminary report describes progress and issues encountered in porting WRF to an HPC cluster running Windows, using Microsoft Subsystem for UNIX-based Applications (SUA) and the PGI Fortran compiler.

Get the complete whitepaper @ http://archives.windowshpc.net/files/699/download.aspx

Note to the following entries

MichaelCole — Fri, 18 Jan 2008 13:57:00 GMT

The following posts are exact copies of entries to the Applications Blog on the previous Windowshpc portal site. I added the author's name in the beginning of each entry and edited the posting dates to reflect the original date of the entry.

Michael Cole
Project Manager/Site Administrator

License aware job application filter

Xpillon — Sat, 19 May 2007 18:54:00 GMT

When many HPC applications have to be used on a cluster, most of the time they are using the well know FlexLM license manager to provide floating licenses. The CCS Job scheduler doesn't do, by default, a license aware scheduling, but by extending it with a Job Activation Filter this feature can be achieved.

This is the goal of the filter associated with this blog for which the documentation is attached.

Quickly speaking, the filter use the license feature term associated to a job in its properties, the with this feature retrieve the license manager to query and check if there are available tokens before activating the job.

Adding license feature can be done thru the /license CLI option or with the license tab in the job properties within the Job Manager.

I hope you will enjoy the filter

Xavier

Attachment(s): LicenseActivationFilter.zip

Fun and interesting ways to run MPI jobs on CCS

MichaelCole — Tue, 03 Oct 2006 13:06:00 GMT

posted Tuesday, October 03, 2006 6:07 PM by elantz | 0 Comments

MS-MPI is identical to MPICH2 in most respects but one area of difference is MS-MPI’s secure execution and interaction with the CCS Job Scheduler. Hopefully, this post will shed some light on how you can leverage the CCS Job Scheduler to run your MPI jobs just exactly the way you like to run them.

Note: The examples in this posting use the CCS Job Scheduler’s Command Line Interface but all these examples work equally well in the Job Manager (Scheduler’s graphical user interface) using the:

· Processors tab: Set the number of processors required (equivalent of /numprocessors: )

· Tasks tab: Enter your mpiexec command

· Advanced tab: Specify specific nodes upon which to run the job (equivalent of /askednodes: )

The Basics

The command to submit an MPI job via the scheduler is of the form:

job submit [specify cluster resources] mpiexec [mpi arguments] Application [app arguments]

Specifying Cluster Resources

Specify the cluster resources to use on your MPI job via one of the following CCS Scheduler Command Line Interface (CLI) arguments:

Resource Specification	Use for Running MPI Jobs
/numprocessors:	Specify the number of processors to be used in for the job. Scheduler will choose nodes to satisfy this processor requirement from either the available compute nodes in the cluster OR the list of compute nodes in /askednodes:. This is the fastest way to get your job running because your job won’t be waiting on specific nodes- you’ve given Scheduler the freedom to run it on the first available processors. NOTE: The default value is 1.
/askednodes:	The Scheduler treats /askednodes: as a “pool” of possible nodes to use until it reaches the numprocessors required for the job. . NOTE: If specified without also specifying /numprocessors:, the job will run on a single processor on a single compute node because the default value of /numprocessors: is 1 !!
/numprocessors: AND /askednodes:	Scheduler will select /numprocessors: from available compute nodes in the /askednodes; list. This is usually the best way to run an MPI job on specific compute nodes.

mpiexec recognizes several command line arguments and one of the most interesting for the purposes of this discussion is –hosts. The –hosts argument is of the form:

-hosts < number of hosts > < host1 name > <number of processes on host1> ... < hostn >

and is a means of specifying the resources to be used for the MPI job. Under normal circumstances you will not specify –hosts as the CCS Scheduler will automatically create this argument in an environment variable (CCP_NODES) which MS-MPI mpiexec uses by default. If you specify a –hosts argument mpiexec will use the one you specify and ignore CCP_NODES. In general, you do NOT want that to happen because it precludes the Scheduler from load balancing your cluster. And since it’s the exception that proves the rule, there are a very few cases where –hosts is useful…some of which are illustrated in examples below.

In general, do NOT use mpiexec’s –hosts argument as it precludes the Scheduler from load balancing the cluster and tends to cause authentication errors if the cluster resource requirements for the job do not match the node list in the –hosts argument.

The Simplest Possible MPI Job

job submit /numprocessors:4 mpiexec MyApp

This command will run the MPI application- MyApp- on the first available 4 processors in the cluster. The job, by default, runs in exclusive mode so the scheduler will not attempt to run any other jobs on the compute nodes chosen. Note that the default for tasks is non-exclusive so that other tasks from the same job can run on a given node. This is why a listing of this job’s environment variable would have included CCP_EXCLUSIVE=false as CCP_EXCLUSIVE refers to the task exclusivity.

Run Your MPI Job Just the Way You Like (Some Examples)

1. Fastest, easiest way to run MyApp.exe on any 5 processors in the cluster and send the output to a shared folder (named “fileshare”) on the headnode
Use /numprocessors: in job submit

1. job submit /numprocessors:5 /stdout:\\headnode\fileshare\out.txt mpiexec MyApp.exe

2. I’ve got to run MyApp on 2 specific compute nodes where I’ve installed special DLL’s
Use /askednodes: & /numprocessors: in job submit

1. job submit /askednodes:Node1,Node2 /numprocessors:3 /stdout:\\headnode\fileshare\out.txt mpiexec MyApp.exe

which, assuming each node has 2 procs, will run MyApp.exe in 2 processes on Node1 and one process on Node2 but other jobs will not be able to use the other Node2 processor because jobs are marked exclusive by default.

3. I’ve got to run MyApp on 2 specific compute nodes where I’ve installed special DLL’s and control the number of processes run on each node
Use /askednodes: and /numprocessors: in job submit and –hosts in mpiexec command

1. job submit /askednodes:Node1,Node2,Node3 /numprocessors:6 /stdout:\\headnode\fileshare\out.txt mpiexec –hosts 3 Node1 2 Node2 2 Node3 1 MyApp.exe

which, assuming each node has 2 procs, will run MyApp.exe on 2 processes on Node1 & Node2 and 1 process on Node3. Note the total number of MPI processes (5) does not equal /numprocessors: (6) and it doesn’t have to. The goal here is to choose a large enough /numprocessors: value so Scheduler will be forced to use all the /askednodes:.

Or,

Use a VB Script (or equivalent) to run MyApp

b. job submit /askednodes:Node1,Node2 /numprocessors:4 /stdout:\\headnode\fileshare\out.txt cscript //Nologo MyScript MyApp.exe

where MyScript will

· Get the name of the application it is to run as a command line argument (MyApp.exe in this case)

· Get the list of nodes that Scheduler assigned to the job by grabbing the CCP_NODES environment variable

· Create and execute a shell command such as “mpiexec –hosts 2 Node1 2 Node2 1 MyApp.exe”

· Pipe STDOUT, STDERR from the child process back to STDOUT, STDERR of the script

· Pass MyApp.exe’s error code back to the error code of the script

4. I’ve got to run MyApp on any 2 compute nodes but I need to run a single process per node because MyApp is multi-threaded and will consume all the procs on the nodes where it runs.
Use /numprocessors: in job submit and either:

2. Submit with a bat file using regular expression substitution to replace the number of procs with “1” for each node in the CCP_NODES environment variable. For example, if all the compute nodes have 4 processors the following simple substitution will work:

mpiexec –hosts %CCP_NODES: 4= 1% MyApp.exe
(note the space before the ‘4’):

Note: This is a hack but works pretty well for simple node naming schemes.

3. The non-hack solution of submitting with a script file like MyScript above which will replace the number of procs with “1” for each node in the CCP_NODES environment variable and run mpiexec

· See the attached file for an example script that does all this and more

4. I want to run MyApp in a process with customer environment variables [MyEnvironmentVariable is set to “CCS_Rocks” and MyEnvVar2 is set to “and Rolls”]
Use the –env argument in mpiexec

1. job submit /numprocessors:4 /stdout:\\headnode\fileshare\out.txt mpiexec –env MyEnvironmentVariable CCS_Rocks –env MyEnvVar2 “and Rolls” MyApp.exe

5. I want to start 2 MPI applications on the same MPI global communicator

1. job submit /askednodes:Node1,Node2 /numprocessors:4 /stdout:\\headnode\fileshare\out.txt mpiexec –hosts 1 Node1 2 MyApp1.exe : -hosts 1 Node2 2 MyApp2.exe

Note1: Assuming 2 procs per node, this command line will start 2 MyApp1 processes on Node1 and 2 MyApp2 processes on Node2 whose ranks will be 0 thru 3 on the same MPI_COMM_WORLD.
Note2: You’ll need to launch mpiexec with a script if want the scheduler to be free to load balance the job to any compute node because you must do some custom tweeking of the two –hosts arguments based on the CCP_NODES environment variable.

A Good Way to Get Into Trouble

1. Specify both /askednodes: in the job submit and –hosts in the mpiexec commands
You’ll have to manually keep the two node lists in sync and any errors will cause an authentication failure.

Documentation of MS-MPI mpiexec Arguments

Full documentation on MS-MPI mpiexec arguments is here: http://technet2.microsoft.com/WindowsServer/en/library/7876c216-b704-473c-b97f-e8a15c67551b1033.mspx?mfr=true

Microsoft Visual Studio Teplate for Application development for Microsoft Windows CCS

MichaelCole — Sun, 04 Jun 2006 13:09:00 GMT

posted Sunday, June 04, 2006 7:15 PM by NileshL | 1 Comments

OK here is a template for Microsoft Visual Studio app development for Microsoft Windows CCS that DennisCr, PhilPen and CaClark helped create and refine.

This template makes it easy for developers who are new to Visual Studio development environment to get started with ease.

The template is designed to be run from a remote Windows computer and will submit a job to a Microsoft Windows Cluster on the network.

Now just insert your code into the template, build an executable…and start submitting jobs to a Microsoft Windows Compute Cluster on your network.

Running MPI ranks/processes on specific nodes

MichaelCole — Thu, 01 Jun 2006 13:11:00 GMT

posted Thursday, June 01, 2006 2:47 PM by NileshL | 0 Comments

We have seen this question come up many times...I recall Ming Xu has already shed light on this in another post in another newsgroup but I think this post will make a bit easier for CCS users.

Scenario:

A user wants to run:

o rank 0 on node h1,

o rank 1 on node h3,

o rank 2 on node h2,

o rank 3 on node h1

Steps to enable that scenario:

C:\> job submit /numprocessors:6 /askednodes:h1,h2,h3 mpiexec -hosts 4 h1 h3 h2 h1 <exe path>

Assumptions:

o The machines are dual-processors boxes

/numprocessors:6 and askednodes:h1,h2,h3 gets three of those machines

o "mpiexec -hosts 4 h1 h3 h2 h1" will place the rank 0 on h1, rank 1 on h3, rank 2 on h2, rank3 on h1

Commercial Applications on Microsoft Windows CCS

MichaelCole — Mon, 29 May 2006 13:13:00 GMT

posted Monday, May 29, 2006 6:23 PM by NileshL | 0 Comments

Just saw Dennis' post where Dennis points to some commercial applications for Microsoft Windows Compute Cluster Server 2003.

I attended ANSYS user conference a few weeks ago in Pittsburgh, PA and I loved what I saw there: ANSYS showcased ANSYS 11.0 solver running on Windows CCS using Microsoft MPI and ANSYS workbench running on Windows XP x64 that invokes ANSYS solver through Remote Solve Manager on a remote cluster running CCS using the Microsoft Job Scheduler.

I am super excited that many different commercial apps from many other ISVs in key industry verticals are ported/in the process of being ported to Windows CCS. I look forward to exciting times in the years ahead of us.

Submitting a simple job to a newly installed CCS

MichaelCole — Sun, 28 May 2006 13:16:00 GMT

posted Sunday, May 28, 2006 11:52 PM by NileshL | 1 Comments

Just finished installing another brand new cluster running the release candidate binaries of CCS I downloaded via links through http://www.microsoft.com/hpc So, the next logical step that comes to mind is to exercise the cluster witha simple application...like the example to calculate PI in parallel using Microsoft MPI. To initiate this PI calculation exercise, I would need to submit a job to the CCS. The CCS includes Microsoft Job Scheduler which provides at least three simple ways of submitting a job to the CCS.

1. via command line

2. using .NET APIs

3. using COM APIs

Although submitting a job via the command line is trivially simple, ala,

c:\workdir> job submit sample.exe

my personal favorite is submitting job through the .NET API using C#. Here is a simple code that I prefer.

The following code submits a "PI.exe" job.
----------------------------------------------
using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.ComputeCluster;

namespace test
{
  class Program
  {
    static void Main(string[] args)
    {
      try
      {
        ICluster cluster = new Cluster();
        cluster.Connect("Nilesh-HeadNode");
        IJob job = new Job();
        ITask task = new Task();
        task.CommandLine = "PI.exe";
        job.AddTask(task);
        cluster.AddJob(job);
        cluster.SubmitJob(job.Id, null, null, true, 0);
      }
      catch (Exception e)
      {
        Console.WriteLine(e.Message);
      }
    }
  }
}

3rd Party Software links

MichaelCole — Wed, 24 May 2006 19:29:00 GMT

posted Wednesday, May 24, 2006 5:29 PM by DennisCr | 0 Comments

Hi there...I thought that we should kick off the application blog with a good list of apps! The list also includes tools and drivers. This list was previously found on the beta web site.

Updated 5/17/06

For general partner information, please see http://www.microsoft.com/hpc.

Tools

o Intel Cluster MKL for Windows CCS beta available at: http://www.intel.com/cd/software/products/asmo-na/eng/cluster/clustermkl/index.htm

o Absoft Fortran v10 for 64-bit WIndows and Windows CCS: http://www.absoft.com/windows64compilerbetasignup.html

o PGI Visual Fortran: http://www.pgroup.com/products/pvf.htm

Applications

o LSTC http://www.lstc.com has info on front page.

o ANSYS http://www1.ansys.com/customer/default.asp requires login credentials

o Wolfram http://www.wolfram.com/products/mathematica/platforms/

o Dassault http://www.3ds.com/products-solutions/plmsolutions/catia/details/#software_requirements

o MathWorks http://www.mathworks.com/products/new_products/latest_features.html has information on x64 support, and http://www.mathworks.com/accesslogin/?uri=web_downloads/page2.html?new=1 for download

Drivers for interconnects:

o Voltaire http://www.voltaire.com/infiniband_windows_stack.htm

o Myricom http://www.myri.com/scs/download-socketsgm.html requires users to request a login ID

o Mellanox http://www.mellanox.com/products/win_ib.php