Process Affinity and Windows HPC Server 2008 SP1

Barndawgie — Thu, 01 Oct 2009 23:57:00 GMT

An issue came to our attention after Windows HPC Server 2008 shipped regarding the way we set affinity on the processes within a job. There are actually two places where we set affinity:

1. The Node Manager (the service on each node responsible for starting jobs and tasks) sets processor affinity on each task to prevent that task from using processors which are not assigned to it.

2. MPIEXEC (which is used to start MS-MPI applications) can, when the –affinity flag is provided, set affinity on all ranks within the MPI application.

The problem that we encountered is this: Due to the way affinity setting works on Windows Job Objects (which we use to run tasks) and the processes within them, you cannot set affinity at both layers. That means that in MPI Tasks which are allocated less than an entire node, the –affinity flag will end up being ignored on the MPIExec command line, since the affinity has already been set by the scheduler and cannot be set in two places. This caused problems for some applications, especially those developed to work against the Compute Cluster Pack (which didn’t set affinity at all).

The problem is particular serious for jobs which specify the –exclusive option; when a job specifies the –Exclusive option it will be allocated an entire node. But the scheduler will set affinity on tasks within the job despite this. So an exclusive job with a 4 core task that is assigned an 8 core node would cause the scheduler to affinitize the task to only 4 cores: This leaves the other 4 cores idle if there are no other tasks in the job and is awfully confusing for some people and applications! Such a job would also not have MPI rank affinity, even if the –Affinity flag was specified.

Our solution is to introduce a new cluster parameter called AffinityType. AffinityType has three possible settings which work as follows:

· AllJobs- When AffinityType is set to AllJobs, the Node Manager will set affinity on any task that isn’t allocated an entire node. This is the behavior described above, and is probably the best choice for applications which may run multiple instances per node (e.g. Parameter Sweeps and SOA Jobs) and want these instances to be isolated from each other.

· NonExclusiveJobs (Default)- With this setting, the Node Manager will not set affinity on jobs which are marked as exclusive. This is the ideal choice for jobs with only 1 task, since that task will be able to take advantage of all cores on the nodes that it is assigned. We’ve made this the new default since it provides what is generally the preferred behavior for MPI tasks, which are most likely to be sensitive to affinitization. With this choice selected, MPI tasks in exclusive jobs can take advantage of the –Affinity flag to MPI even if they are not allocated an entire node.

· NoJobs- With this setting, the Node Manager will never set affinity on any task. This is an excellent choice for those running MPI Tasks who want to make sure they can take advantage of MPI’s –Affinity flag even when jobs may share nodes. This is also useful for applications which want to set their own affinity.

Note that Windows Server 2008 R2 will allow the setting of process affinity at both the Windows Job Object and Process level simultaneously. So hopefully in v3 of the HPC Pack there will no longer be an issues with the conflict between these two setting.

You can learn more about how the new AffinityMode flag works here: http://msdn.microsoft.com/en-us/library/microsoft.hpc.scheduler.properties.affinitymode(vs.85).aspx

HPC Server 2008 MPI Diagnostic Fails on Eager Message No Business Card Error

FrankChism — Mon, 29 Jun 2009 19:33:00 GMT

An HPC Server 2008 user reported that his cluster was up and running and that all nodes could ping each other over all networks but the built-in MPI diagnostic was failing with an uninformative message "Failed To Run".

He had a topology number three with the head node connected to the Enterprise network and all compute nodes connected to the head node via Ethernet as the Private network and Infiniband as the Applications network.

Please be aware that "Failed To Run" is a separate category from "Failure" and when a test doen't succeed, you may have to check both places in the Diagnostics tree Test Results branch. Once you find this tab you don't get much information beyond the result "Failed To Run". However if you click on the red ! labelled line you will see the bottom pane light up, but it still only says "Test Failed to Run". Look to the right side of that banner and you will see a bright red "Result" followed by a v in a circle. Click on the v and you get more information about the failure.

The test did not run. Please navigate to 'Progress of the test' to view log and error messages.

So where is the Progress of the test to be found? Well, if like me you often don't have the Actions pane open, you better click on the Actions tab near the top of the console. Now near the top of the Action pane you will see the link to "Progress of the Test". This is progress, of a sort. You'll likely see just a single line with the red ! and a State of "Reverted". Now click on that line.

Oh, boy, we're rockin' now. Here are the real error messages. This is so information rich it's almost embarassing.

Time Message
6/29/2009 10:16:53 AM Reverted
6/29/2009 10:16:53 AM The operation failed due to errors during execution.
6/29/2009 10:16:53 AM The operation failed and will not be retried.
6/29/2009 10:16:53 AM ---- error analysis -----
6/29/2009 10:16:53 AM
6/29/2009 10:16:53 AM mpi has detected a fatal error and aborted mpipingpong.exe
6/29/2009 10:16:53 AM [2] on NODE-03
6/29/2009 10:16:53 AM
6/29/2009 10:16:53 AM ---- error analysis -----
6/29/2009 10:16:53 AM
6/29/2009 10:16:53 AM [3-6] terminated
6/29/2009 10:16:53 AM
6/29/2009 10:16:53 AM Check the local NetworkDirect configuration or set the MPICH_ND_ENABLE_FALLBACK environment variable to true.
6/29/2009 10:16:53 AM There is no matching NetworkDirect adapter and fallback to the socket interconnect is disabled.
6/29/2009 10:16:53 AM CH3_ND::CEnvironment::Connect(296): [ch3:nd] Could not connect via NetworkDirect to rank 1 with business card (port=58550 description="10.1.0.2 192.168.0.28 192.168.0.39 NODE-02 " shm_host=NODE-02 shm_queue=3204:428 nd_host="10.1.0.2:157 " ).
6/29/2009 10:16:53 AM MPIDI_CH3I_VC_post_connect(426)...: MPIDI_CH3I_Nd_connect failed in VC_post_connect
6/29/2009 10:16:53 AM MPIDI_CH3_iSendv(239).............:
6/29/2009 10:16:53 AM MPIDI_EagerContigIsend(519).......: failure occurred while attempting to send an eager message
6/29/2009 10:16:53 AM MPIC_Sendrecv(120)................:
6/29/2009 10:16:53 AM MPIR_Allgather(487)...............:
6/29/2009 10:16:53 AM MPI_Allgather(864)................: MPI_Allgather(sbuf=0x00000000001FF790, scount=128, MPI_CHAR, rbuf=0x0000000000B70780, rcount=128, MPI_CHAR, MPI_COMM_WORLD) failed
6/29/2009 10:16:53 AM Fatal error in MPI_Allgather: Other MPI error, error stack:
6/29/2009 10:16:53 AM [2] fatal error
6/29/2009 10:16:53 AM
6/29/2009 10:16:53 AM [0-1] terminated
6/29/2009 10:16:53 AM
6/29/2009 10:16:53 AM [ranks] message
6/29/2009 10:16:53 AM job aborted:
6/29/2009 10:16:53 AM

First it tells us that Node-03 had a problem. Then it tells us to look at the Node-03 local Network Direct connection. Then it tells us that the environment is set to not fall back to Winsock Direct or TCP/IP. This is because falling back when people are expecting Network Direct performance can cause applications to run very slowly and is hard to diagnose. Trust me. I missed sleep over that one.

Then we have serveral lines of MPI error messages, which I generally summarize as the 'eager message no business card' error. You can ignore the rest of the message but keep in mind that whenever you see the eager message no business card error you should suspect your MPI network has a problem.

So, let's follow the advice at the beginning of the error messages, and check the InfiniBand status on Node-03. I use the Run Command feature of the Management Console to run the ndinstall tool. To make life easier, I copy the .exe for this tool to all of the nodes in C:\Windows\System32\ndinstall.exe . This tool is usually installed by the .msi install of the drivers on the head node. Search your system drive after you install the drivers and find this tool. Then put it on a head node share the compute nodes can see and use clusrun or the Run Command GUI to copy it to all the compute nodes. Here's the output from Node-03 (bad node no business card) and Node-02 (good node, pat pat).

Node-03

0000001001 - MSAFD Tcpip [TCP/IP]
0000001002 - MSAFD Tcpip [UDP/IP]
0000001003 - MSAFD Tcpip [RAW/IP]
0000001004 - MSAFD Tcpip [TCP/IPv6]
0000001005 - MSAFD Tcpip [UDP/IPv6]
0000001006 - MSAFD Tcpip [RAW/IPv6]
0000001007 - RSVP TCPv6 Service Provider
0000001008 - RSVP TCP Service Provider
0000001009 - RSVP UDPv6 Service Provider
0000001010 - RSVP UDP Service Provider

Node-02

Notice there is no 0000001011 OpenIB Network Direct Provider on Node-03. So, actually the diagnostic got it right immediately. I just took a long time to prove it. So now let's run ndinstall -i on Node-03. Again with the Run Command, eh? All we get is a Finished. Then run ndinstall -l again and verify that we get the 0000001011 OpenIB Network Direct Provider line. Yes we do, but do not be confused if, for you like for me, it is "0000001012 - OpenIB Network Direct Provider". The sequence number is not important.

And finally let's run the diagnostic again and look in Diagnostics->Test Results->Success. Ah, now that's sweet!

Test Name Result Test Suite Target Last Updated
MPI Ping-Pong: Quick Check Success Performance 7 nodes 6/29/2009 10:54:58 AM

That's it for now, "Transfer fast and prosper."

Frankie

The Windows HPC Team Blog : MSMPI HPC Server InfiniBand Built-in Diagnostics

Process Affinity and Windows HPC Server 2008 SP1

HPC Server 2008 MPI Diagnostic Fails on Eager Message No Business Card Error