The Windows HPC Team Blog : Job Scheduling

Process Affinity and Windows HPC Server 2008 SP1

Barndawgie — Thu, 01 Oct 2009 23:57:00 GMT

An issue came to our attention after Windows HPC Server 2008 shipped regarding the way we set affinity on the processes within a job. There are actually two places where we set affinity:

1. The Node Manager (the service on each node responsible for starting jobs and tasks) sets processor affinity on each task to prevent that task from using processors which are not assigned to it.

2. MPIEXEC (which is used to start MS-MPI applications) can, when the –affinity flag is provided, set affinity on all ranks within the MPI application.

The problem that we encountered is this: Due to the way affinity setting works on Windows Job Objects (which we use to run tasks) and the processes within them, you cannot set affinity at both layers. That means that in MPI Tasks which are allocated less than an entire node, the –affinity flag will end up being ignored on the MPIExec command line, since the affinity has already been set by the scheduler and cannot be set in two places. This caused problems for some applications, especially those developed to work against the Compute Cluster Pack (which didn’t set affinity at all).

The problem is particular serious for jobs which specify the –exclusive option; when a job specifies the –Exclusive option it will be allocated an entire node. But the scheduler will set affinity on tasks within the job despite this. So an exclusive job with a 4 core task that is assigned an 8 core node would cause the scheduler to affinitize the task to only 4 cores: This leaves the other 4 cores idle if there are no other tasks in the job and is awfully confusing for some people and applications! Such a job would also not have MPI rank affinity, even if the –Affinity flag was specified.

Our solution is to introduce a new cluster parameter called AffinityType. AffinityType has three possible settings which work as follows:

· AllJobs- When AffinityType is set to AllJobs, the Node Manager will set affinity on any task that isn’t allocated an entire node. This is the behavior described above, and is probably the best choice for applications which may run multiple instances per node (e.g. Parameter Sweeps and SOA Jobs) and want these instances to be isolated from each other.

· NonExclusiveJobs (Default)- With this setting, the Node Manager will not set affinity on jobs which are marked as exclusive. This is the ideal choice for jobs with only 1 task, since that task will be able to take advantage of all cores on the nodes that it is assigned. We’ve made this the new default since it provides what is generally the preferred behavior for MPI tasks, which are most likely to be sensitive to affinitization. With this choice selected, MPI tasks in exclusive jobs can take advantage of the –Affinity flag to MPI even if they are not allocated an entire node.

· NoJobs- With this setting, the Node Manager will never set affinity on any task. This is an excellent choice for those running MPI Tasks who want to make sure they can take advantage of MPI’s –Affinity flag even when jobs may share nodes. This is also useful for applications which want to set their own affinity.

Note that Windows Server 2008 R2 will allow the setting of process affinity at both the Windows Job Object and Process level simultaneously. So hopefully in v3 of the HPC Pack there will no longer be an issues with the conflict between these two setting.

You can learn more about how the new AffinityMode flag works here: http://msdn.microsoft.com/en-us/library/microsoft.hpc.scheduler.properties.affinitymode(vs.85).aspx

Issue with HPC Event Logs with HPC Pack SP1

Barndawgie — Fri, 18 Sep 2009 23:31:00 GMT

Some customers have reported issues with the HPC Job Scheduler's event logs after installing Service Pack 1 for the HPC Pack. After some internal investigation, it looks like there is an issue with the SP1 Patch Installer which fails to correctly update the event log manifests (making it so you don't see events in the Event Viewer).

We are working on a patch for this issue. In the meantime, you can manually update the manifest as follows:

Download the file attached to this blog post (SchedulerEvents.man)
Install this manifest on the affected machine by running the following from an elevated command prompt: "wevtutil im SchedulerEvents.man"

That's it! This workaround should restore event log functionality. We are hard at work on a patch that will make this workaround obsolete.

UPDATE: We've managed to fix our patch installer to correct this problem. Installing any post-SP1 patch (for example, this one: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=9d76e266-92d1-49b0-8bf3-cd811b6a5a4c) to the HPC Pack should fix this issue if you encounter it.

Dryad and DryadLINQ on HPC Server

JVert — Wed, 15 Jul 2009 23:04:00 GMT

Microsoft Research has an interesting project called Dryad which is investigating programming models for distributed data-parallel problems. If you've heard some of the hype around map/reduce programming or the Hadoop framework, you might be familiar with these type of problems. If you have gigabytes, terabytes, or even petabytes of data that you need to churn through, this is a powerful tool. On top of the Dryad framework is a programming model called DryadLinq which brings the power and familiarity of .NET's LINQ syntax to cluster computing. You can write your data analytics algorithm as a LINQ expression in your favorite .NET language and submit cluster jobs directly from Visual Studio.

I am delighted to announce that Microsoft Research has released a distribution of Dryad and DryadLINQ that runs on top of HPC Server 2008. You can download this release from here and contribute feedback and suggestions at the Microsoft Connect site here. If you want to learn more about Dryad and DryadLINQ, you can find more information on the Microsoft Research site. The source for DryadLINQ is also included in this release.

Releasing this has been a great collaboration between the HPC team and Microsoft Research. I am excited to get this work out there and see what people can do with it. So try it out and give us some feedback!

John Vert

Architect

Windows HPC Server

Update - Channel9 has a video interview with Erik Meijer and Roger Barga discussing this: http://channel9.msdn.com/posts/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ/

Submit job from SUA (Subsystem for Unix based Application)

Thomas.varlet@MS — Wed, 06 May 2009 19:00:00 GMT

SUA is a very nice tool for porting Unix/Linux application, but it can also be used to submit job. You just have to set the path to the HPCS binary into your PATH variable :

With BASH :
export PATH=$PATH:"/dev/fs/C/Program Files/Microsoft HPC Pack/Bin"

With Korn Shell :
set PATH $PATH:"/dev/fs/C/Program Files/Microsoft HPC Pack/Bin"

And then you can submit job (copy from a Korn Shell windows on my head node):

$ set PATH $PATH:"/dev/fs/C/Program Files/Microsoft HPC Pack/Bin"
$   job.exe submit /workdir:\\\\hpcs-fr\\Shared /stdout:out-tv.txt /stderr:ett-tv.txt ping -n 50 hpcs-fr
Job has been submitted. ID: 37.
$ job.exe list
Id         Owner                Name                                     State        Priority    Resource Request
---------- -------------------- ---------------------------------------- ------------ ----------- ------------------
36         HPCSFR\Administrator                                          Running      Normal      *-* cores
37         HPCSFR\Administrator                                          Running      Normal      *-* cores

2 matching jobs for HPCSFR\administrator

How to set environment variables on computes nodes

Thomas.varlet@MS — Tue, 07 Apr 2009 21:59:00 GMT

For ISV application you have to set some environments variables on the compute node.
You can set this variable by hand, but if you have many nodes it can be long.
So, the idea of this post is how can set them attaumaticaly ?

Use the HPCS tools :

You can set cluster wilde variable by using cluscgf in a Administrator cmd line windows :
"cluscfg setenvs name=value"

ex. : cluscfg setenvs LM_LICENSE_FILE=license_server@74000

So each time you will submit a job this variable will be set for the job

Use command line :

Select the nodes into the node management view and run the following command
"setx name value /M"
This command will add this environement variable in the system of all the selected nodes

NOTE : /M set the variable for the system

ex. : setx PATH "%PATH%;\\headnode\Apps\bin" /M

Troubleshooting Windows HPC WCF/SOA Issues

ihimmar — Fri, 03 Apr 2009 12:41:00 GMT

HPC uses HPC sessions to support the service-oriented architecture (SOA) programming model based on Windows Communication Foundation (WCF). Sometimes troubleshooting errors from this SOA based applications could be challenging. However this tip I'm about to share should be helpful to figure out exactly where the issue is coming from. Looking through the trace of the communication between the service hosts (running on the compute nodes) and the broker is often the key to identifying where the problem lies.

You can use the Windows Communication Foundation (WCF) Service Trace Viewer Tool to analyze messages logged by WCF. Service Trace Viewer is included in the Microsoft Windows Software Development Kit (SDK) for Windows Vista and .NET Framework Runtime Components. You can download the Windows SDK from the Microsoft Download Center at http://go.microsoft.com/fwlink/?LinkID=75636. For more information about using this tool, see "Service Trace Viewer Tool (SvcTraceViewer.exe)"at http://go.microsoft.com/fwlink/?LinkId=88991.

The following are the instructions to enable tracing.

1. Modify the system.diagnostic section HpcServiceHost.exe.config in %CCP_HOME%\bin\ (*for each compute nodes*) as follow:

<system.diagnostics>

</listeners>

</source>

</sources>

<add initializeData="\\<HEADNODE>\CcpSpoolDir\host.svclog" type="System.Diagnostics.XmlWriterTraceListener"

name="ServiceHostTraceListener">

</add>

<add type="System.Diagnostics.ConsoleTraceListener, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"

name="Console" traceOutputOptions="DateTime, ThreadId">

</add>

</sharedListeners>

</system.diagnostics>

Modify the modify the system.diagnostic section of the HpcWcfBroker.exe.config in %CCP_HOME%\bin (*on all broker nodes*) as follows:

<system.diagnostics>

</add>

</add>

</listeners>

</source>

</sources>

<add type="System.Diagnostics.ConsoleTraceListener, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"

name="Console" traceOutputOptions="DateTime, ThreadId">

</add>

<add initializeData="\\<HEADNODE>\CcpSpoolDir\broker.svclog"

type="System.Diagnostics.XmlWriterTraceListener, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"

name="WSLBTraceListener" traceOutputOptions="Timestamp">

</add>

</sharedListeners>

</system.diagnostics>

2. replace the <HEADNODE> in both files with your headnode name.

3. Run your application until you see the errors

4. All svclog files will be under \\<HEADNODE>\CcpSpoolDir\.

a powershell problem

Barndawgie — Wed, 07 Jan 2009 04:22:00 GMT

We've gotten reports of an installation glitch that makes it so that node management cmdlets won't work when run from a remote client (they'll work just fine run on the headnode) due to a registry key that doesn't always get set up correctly. Fortunately, there is a simple work around! If you are trying to run PowerShell from a remote client and get an error like this:

PS> get-hpcnode
Get-HpcNode : Unable to find assembly 'Microsoft.Ccp.Sdm.Store, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null'.
At line:1 char:12
+ get-hpcnode <<<<
    + CategoryInfo          : NotSpecified: (Microsoft.ComputeCluster.CCPPSH.GetNode:GetNode) [Get-HpcNode], Serializa
   tionException
    + FullyQualifiedErrorId : Microsoft.ComputeCluster.CCPPSH.GetNode

You should be able to remedy the problem by adding the following registry key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\HPC\BinDir" with a value of "C:\Program Files\Microsoft HPC Pack\Bin\". You can either do so manually, or just run the .REG file attached to this post.

Then, just open a new PowerShell window and you should be good to go! We hope to have a fix for this at some point soon.

7 Minute Job Scheduling Thrill-Ride

Barndawgie — Thu, 27 Nov 2008 01:18:00 GMT

Shahrokh was nice enough to film and post a video of me walking through the HPC Job Scheduler and some of its new features at SuperComputing 2008. You can check out the video up on YouTube.

In the video I cover many of the different scheduler interfaces and policies, so it's a great primer for those unsure about what our scheduler offers.

Creating Submission and Activation Filters

Barndawgie — Tue, 11 Nov 2008 03:54:00 GMT

There were a lot of questions about how to use Activation and Submission Filters to help customize queue managmeent and do things like license-aware scheduling. That, on top of some changes made in a QFE, led us to do an updated doc on using filters. You can check it out here:

http://technet.microsoft.com/en-us/library/dd277833.aspx

It contains updated sample code and explanations. Let us know if you think any information is missing!

Integrating Windows HPC Server 2008 with Linux

StevenN — Thu, 02 Oct 2008 12:51:00 GMT

We find many of our Windows HPC Server 2008 deployments are going into environments where there are existing Linux (and Linux HPC) solutions. It is possible to configure these two environments to achieve integrated authentication, file sharing and job submission.

The following technical documents provide step by step instructions on how to do a typical installation of Windows HPC and a Linux HPC distribution so as to achieve a single sign on environment, file sharing and to submit jobs from a Linux environment running Sun Grid Engine into the Windows HPC 2008 job scheduler using the HPC Basic Profile web service specification.

Title	Details Page	Direct Download
Installation of Fedora Samba for Windows AD Compatibility	http://www.microsoft.com/downloads/details.aspx?FamilyId=1C2C91A8-6D81-4BC2-94E9-448D68A7D06D&displaylang=en	http://download.microsoft.com/download/1/6/9/16963418-6d06-4cb6-8b65-9fe3da11c583/Installation_of_Fedora-Samba_for_Windows_AD_Compatibility_Final.doc
Installation Instructions for Cluster Corp Rocks+ on the HP Proliant DL145 G2 Based Cluster	http://www.microsoft.com/downloads/details.aspx?FamilyId=7AE6D41D-4C86-4B34-9C62-466646915926&displaylang=en	http://download.microsoft.com/download/5/b/5/5b55533c-12a6-4979-8849-f7b7a57eff61/Linux_Installation_Final.doc
Installation of Fedora 8 Linux to Access a Windows HPC Server 2008 Cluster	http://www.microsoft.com/downloads/details.aspx?FamilyId=25647D7B-CE6A-45D2-8472-12F4DB537951&displaylang=en	http://download.microsoft.com/download/f/1/9/f190260a-a892-491b-ab2e-c884c72a9e7d/Linux_Installation_for_Windows_Cluster_Access_Final.doc
Installation Instructions for a Windows HPC Server 2008 Based Cluster on HP Proliant DL145 G2 Hardware	http://www.microsoft.com/downloads/details.aspx?FamilyId=1349438A-A05B-4E2B-91F8-8BF3058EB307&displaylang=en	http://download.microsoft.com/download/0/1/a/01af1aba-0015-4236-a4ab-7498d2e51829/Microsoft_Installation_with_AD_Final.doc
The Windows HPC Server 2008 Cluster in a Linux Environment	http://www.microsoft.com/downloads/details.aspx?FamilyId=9E65676E-D34E-4671-B841-0D1DCA996A8B&displaylang=en	http://download.microsoft.com/download/1/1/6/116be099-9c6b-424c-81e6-c9ce2455ae80/Windows HPC in Linux Environment_Final.doc

Clusrunning with Windows HPC Server 2008

Barndawgie — Wed, 17 Sep 2008 14:28:00 GMT

One of our most popular features in the Compute Cluster Pack was clusrun (known to you GUI users as “Remote Command Execution”), which allowed you to run a command line command across a set of cluster nodes in parallel, with their output piped back to you on the client. Not content to rest on our laurels, we’ve made some additions to clusrun’s capabilities in Windows HPC Server 2008. I’ll dig into some of them below.

But First, Clusrun Basics

At a basic level, clusrun runs a job with a task in it for each node that you specify. This job completely bypasses the queue to start right away, and the tasks pipe their information back to the client machine. This has a couple of requirements to work, namely:

· All of the target machines must be nodes in the cluster (with the HPC pack installed and able to communicate with the head node), but they don’t have to be in the “Online” state

· Your compute nodes must be able to right to a fileshare on the client computer; you can test this by logging into a node and attempting to connect to \\client\c$

· Your job scheduler needs to be working

Assuming these requirements are met, you can run a clusrun command either from the command line (using the clusrun command) or from the HPC Cluster Manager (by right clicking some nodes and selecting “Run Command . . .”). As a simple example, try running clusrun /all hostname.exe, each of the nodes in your cluster will print out its name onto your client:

PS> clusrun /all hostname.exe

Enter the password for 'REDMOND\jbarnard' to connect to 'JBarnardHN':

Remember this password? (Y/N)Y

-------------------------- JBARNARDCN01 returns 0 --------------------------

JBARNARDCN01

-------------------------- JBARNARDCN03 returns 0 --------------------------

JBARNARDCN03

-------------------------- JBARNARDHN returns 0 --------------------------

JBarnardHN

-------------------------- JBARNARDCN02 returns 0 --------------------------

JBARNARDCN02

-------------------------- Summary --------------------------

4 Nodes succeeded

0 Nodes failed

So What’s New?

There are a lot of new options for clusrun in HPCS 2008. These includes

New Formatting Options: Sorted or Interleaved Output

By default, clusrun returns output as each node completes the command. But you can override this by using either the /sorted or /interleaved flags.

/Sorted prints node output in alphabetical order, making it easier to find a specific node. /Interleaved prints out lines of output as they come back, which is great for processing with a script or for determining just where things are going wrong.

Picking Your Nodes: Exclude, Job, Task

We’ve got some great new options for picking your nodes, including the ability to exclude a set of nodes with the /exclude flag. So the command “clusrun /all /exclude:Node14 ipconfig” will return the IP configuration of every node other than Node14.

Next up are the /job and /task options, which are my personal favorites! They allow you to run a clusrun command against all of the nodes which are (or were) assigned to a particular job or task. For example, “clusrun /task:10.4 del /q SomeFile.txt” will delete SomeFile.txt from every node that ran task #10.4.

History Tracking

Clusrun jobs now live in the database just like regular jobs, making it easier to track what you’ve done and to uncover failures. You can easily find them from the command line by running job list /jobname:”Remote command”, or in the HPC Cluster Manager by selecting the “Clusrun Commands” node in the navigation pane. Each node in the run will have a separate task (including exit code, error message, etc . . .) allowing you to more easily dig into the causes of failures.

Happy Clusrunning!

-Josh

MPI Process Placement with Windows HPC Server 2008

Barndawgie — Tue, 16 Sep 2008 19:37:00 GMT

We get an awful lot of questions about how to go about getting the desired process placement across nodes in an MPI job (just check our forums if you don’t believe me), so I thought I’d post here to shed some light on the things that are possible.

Requesting Resources

The first thing you need to do is express the number of resources that you’d like allocated for your job. This can be done at either the node, socket, or core, level. Requesting 4-8 cores will assign your job as few as 4 and as many as 8 processors; requesting 4-8 nodes will get you as few as 4 and as many as 8 hosts, and so forth. For more details on how this sort of request works, check out my previous post, How that Node/Socket/Core thing Works.

Why a Min and a Max?

You may be wondering why it’s necessary to specify a Min and a Max for your job. The reason is that this enables you to help the Job Scheduler decide when and where to run your job. The job scheduler will start your job as soon as it has at least the minimum number of processors. If at that time there are more than the minimum number of resources available, the scheduler will give you up to your maximum number of resources. Thus, setting a small min will allow your job to start sooner, while setting a large min will cause your job to wait in the queue until more resources are available. There is no best way for everyone: you should decide on the min and max that works best for you! But the general guidelines are:

· The smaller your Minimum, the sooner your job will run, so pick as small a Minimum as you’d reasonably accept

· Set a Maximum that’s as large as your job can take advantage of to reduce its overall run time, so pick a Maximum which matches the largest your application scales with reasonable performance gains

· You can always set the Minimum and Maximum to the same value to request a fixed number of nodes

This capability, and all of the MPI process placement features described below, are designed to allow you to specify how you want your job to run without needing to know ahead of time how many or which nodes your MPI application will end up running on.

Setting How Many Processes and Where They Will Run

Once you’ve figured out how many resources you want for your job, the next step is to figure out how you want your MPI ranks started across these nodes. By default, ranks will be started on a “1-per resource requested” basis, so requesting 4 sockets will result in 4 MPI ranks, 1 per socket. For example:

Command Line

Result

C:\>job submit /numsockets:9 mpiexec MyApp.exe

9 ranks of MyApp.exe will be started across an unknown number of nodes, with no more than 1 rank per socket started on any node.

C:\>job submit /numnodes:2-4 mpiexec MyApp.exe

2-4 ranks of MyApp.exe will be started across 2-4 physical nodes (depending on how many are available), with 1 rank per node.

Table 1: Submitting an MPI Job with Default Process Placement

Setting the Number of Cores per Node with -c

We provide a new mpiexec option, -cores (or –c) which allows you to specify the number of ranks to start on each node assigned to your job. This especially useful with node-level scheduling; allowing you to control the size and placement of your job with laser-like precision! Adding some of the other node selection options (like corespernode) will make this even more powerful. For example:

Command Line	Result
C:\>job submit /numnodes:4-4 mpiexec –cores 2 MyApp.exe	MyApp.exe will be started across 4 nodes, with 2 ranks per node (for a total of 8 ranks).
C:\>job submit /numnodes:1-8 mpiexec –cores 3 MyApp.exe	Between 3 and 24 ranks of MyApp.exe will be started, with 3 ranks per node spanning up to 8 nodes.
C:\>job submit /numnodes:8 /corespernode:8 mpiexec –cores 7 MyApp.exe	MyApp.exe will start on 8 nodes. All 8 nodes must have at least 8 cores on them, and 7 ranks of MyApp.exe will be started on each of the nodes (for a total of 56 ranks).

Table 2: Submitting an MPI Job and Specifying the Number of Cores per Node

Note: The /corespernode option refers to the minimum number of cores which must be present on any node assigned to the job, not the number of cores to allocate on a node.

Setting the Number of Total Ranks with -n

You can use the –n argument to mpiexec to set the total number of ranks to start across the entire run, allowing even more fine grained control. For example:

Command Line

Result

C:\>job submit /numcores:8 mpiexec –n 16 MyApp.exe

16 ranks of MyApp.exe will be started, 2 to a core over 8 cores.

C:\>job submit /numnodes:4 mpiexec –n 8 MyApp.exe

8 ranks of MyApp.exe will be started across 4 nodes.

Table 3: Using the -n option to mpiexec

Now Set Affinity

Setting affinity can result in huge performance improvements for MPI applications, and we’ve made it way easier for you to take advantage of affinity! How easy? Just use the –affinity flag to mpiexec, and each rank of your MPI application will be locked to a single core (which can dramatically improve performance for certain applications). For example:

Command Line

Result

C:\>job submit /numnodes:2-4 mpiexec –cores 2 –affinity MyApp.exe

MyApp.exe will be started with 2 ranks on each of between 2 – 4 nodes, for a total of 4 -8 ranks. Each rank will get affinity to one of the cores on its assigned node, so that the two ranks sharing a node cannot step on each other’s toes. If the nodes in question have a NUMA architecture, the ranks on each node will automatically be placed on separate NUMA nodes.

C:\>job submit /numsockets:8 mpiexec –affinity MyApp.exe

8 ranks of MyApp.exe will be started across an unknown number of nodes, where each rank will have a dedicated path to memory that cannot be used by any other job.

Table 4: Submitting an MPI Job with Affinity

Note: Mpiexec will attempt to automatically insure that ranks are spaced as “far apart” as possible; i.e. on different sockets in a NUMA system.

Checking Your Work

You can run a very simple test that will tell you how your placement worked. If you use mpiexec –l hostname.exe (plus any other arguments you need), your output will be a list of MPI ranks and the nodes that they appeared in. This will allow you to see the number of ranks started on each node, as well as the round-robin order that mpiexec uses.

Go Forth and Conquer!

That’s the scoop on process placement with Windows HPC Server 2008. Go try it out! And if you encounter any problems, please post up on our forums and we’ll be happy to help you out.

How to Submit a Job

Barndawgie — Thu, 31 Jul 2008 19:33:00 GMT

I was asked today where there was a quick explanation of how to submit a job from the command line, and I realized that I didn't have a really good answer. So I figured I'd do a blog post (first in a while, I know!) on how to do this.

Starting Simple

Let's say your have an application called "Divide.exe" which takes two arguments, "-Numerator X" and "-Denominator Y". So for example, to divide 3/6 you would run the command:

divide.exe –Numerator 3 –Denominator 6

So now you want to run this thing on a single processor on some node in the cluster. It's really easy! You can just do:

job submit divide.exe –Numerator 3 –Denominator 6

No problem, right?

Getting Parallel

Of course if you're using a cluster, you don't just want to run one command line on one processor of one machine. You want to run in parallel! There are a few ways that you can create parallel jobs depending on the way that your application works.

The two most common ways to do this in HPC are using MPI or using Parameter Sweeps.

Submitting MPI Jobs

Submitting MPI jobs is just as easy as submitting any other job! You simply do a job submit followed by your mpi command line. The number of options offered by mpi is astounding (try mpiexec –help3 for more on that) and will probably be covered in a future post, but the basic command line to submit a 16 processor mpi job would be:

Job submit /numcores:16 mpixec MpiDivide.exe –Numerator 3 –Denominator 6

Submitting a Parameter Sweep

We use the term "parameter sweep" to refer to a job which encompasses running a single, serial application many, achieving parallelism by running many instances of it at the same time. In HPC Server (and in the Compute Cluster Pack), this is accomplished by creating a job with many tasks. So for example, you could create a job that divded 3 by 6, 9, and 12 by doing the following (where X is the job ID returned by the first step):

Job new /jobname:"My Parameter Sweep Job"

Job add X divide.exe –Numerator 3 –denominator 6

Job add X divide.exe –Numerator 3 –denominator 9

Job add X divide.exe –Numerator 3 –denominator 12

Job submit /id: X

New in HPC Server 2008, we have a much simpler (and faster!) way of running parameter sweeps using wildcards! This approach will save you a lot of time since they're much easier to create and because the Job Scheduler needs to store much less information for a Parametric Task than it does for job with many distinct tasks. An example of how to do the same step above using this new technique follows:

Job submit /parametric:6-12:3 divide.exe –Numerator 3 –denominator *

For more info on how to use the new HPC Server parametric sweeps, see my earlier blog post, Making a Clean Sweep with Windows HPC Server 2008.

For more info on the command line tools in general, check out our Command Line Reference. Unfortunately it's not yet updated for v2 . . . but there should be a new one out on the web in just a few weeks!

Job Template White Paper is Now Available

Barndawgie — Thu, 19 Jun 2008 10:47:00 GMT

We've posted a white paper on how Job Templates can help you manage your cluster. You can check out the white paper here:

https://windowshpc.net/Blogs/jobscheduler

Making a Clean Sweep with Windows HPC Server 2008

Barndawgie — Thu, 15 May 2008 21:11:00 GMT

Parameter Sweeps are one of the most common types of jobs that get run on HPC clusters, and we've done some work in Windows HPC Server 2008 to make them easier (and faster) than ever. Today I'll dig in a little on what's different and how to take advantage of these new features.

For those familiar with other scheduling products, you may recognize these as very similar to "Job Arrays."

What is a Parameter Sweep?

Simply put, a parameter sweep is when you run a single command many times over a set of different input parameters. Problems which can be solved in this way are very common. They're also a great use of HPC clusters, since parameter sweeps are inherently embarrassingly parallel; namely they can be run in parallel with little or no effort and scale almost limitlessly.

In general, Parameter Sweeps take the form of a single command line which is run N times, with different input, output, and/or command line arguments for each of the N steps. For example, think about an application called FileZipper.exe that takes an input file and generates a compressed output file. To run it on 100 data files, you'd basically want to do something like:

FileZipper.exe <FileToZip1.dat >ZippedFile1.zip

FileZipper.exe <FileToZip2.dat >ZippedFile2.zip

FileZipper.exe <FileToZip3.dat >ZippedFile3.zip

…

FileZipper.exe <FileToZip100.dat >ZippedFile100.zip

These instances can already independently of one another, and you can run as many in parallel as you have processors to do the compression with.

Make sense? If it does, then you already understand pretty much everything there is to know about Parameter Sweeps.

What's different in Windows HPC Server 2008?

In the Compute Cluster Pack (our product from 2005), parameter sweeps could be generated pretty easily from the UI by inputting the Start Index (the number to start counting from), End Index (the last number to use use), and the Increment (the number to add for each step in the sweep). Using the UI to create such a sweep would then create N individual tasks in the job. This was useful, but had a number of downsides, namely:

You're storing a lot of repeated information in the scheduler database
If you wanted to change a part of your sweep, you had to change every task
Large sweeps became very unwieldy to view in the UI or at the command line

In Windows HPC Server 2008, we add a new type of task called a Parametric Task:

Figure 1: Adding a Parametric Task to a Job

Now, these tasks are stored as a unit, which makes, storing, editing, managing, and monitoring them a snap! Let's go ahead and give it a try . . .

Creating a Parameter Sweep

To create your first parameter sweep, open up the HPC Job Manager. The simplest way to create a sweep is to click on the Parametric Sweep Job link in the right-hand Actions Pane.

Let's create a sweep in the form of the example up above; namely, a sweep that zips up 100 files:

1. Go ahead and provide a Name like "File Zipper".

2. Set the Start Value of 1 and an End Value of 2.

3. Leave the Increment Value to be 1 (since we'll be counting up by 1's).

4. Enter your Command Line, in this case "FileZipper.exe".

5. We'll see the Standard Input and Standard Output as above:

1. Stdin: FileToZip*.dat

2. Stdout: ZippedFile*.zip

6. Check the preview box at the bottom of the dialog to see what your sweep will look like, then go ahead and submit.

You should end up with something that looks like this:

Figure 2: Creating a Parameter Sweep Task

Tracking Your Sweep's Progress

If you check the job list, you should now see that you've submitted a job with a single task in it. But actually, you can easily track each task individually by checking the box labeled Expand parametric tasks. This allows you track your sweep as a unit, but dig in on failures or results for individual steps.

Doing that From the Command Line

Of course you can do the same thing from the Command Line:

C:\>job submit /parametric:100 /StdIn:"FileToZip*.dat" /StdOut:"ZippedFile*.zip" FileZipper.exe

Or from PowerShell:

PS> New-HpcJob | Add-HpcTask -Parametric -Start 1 -End 100 -Stdin "FileToZip*.dat" -Stdout "ZippedFile*.zip" -CommandLine "FileZipper.exe" | Submit-HpcJob

That's all for this time. Happy sweeping!