CCS v1 - How to setup network adapters and bindings on Compute nodes ?

tomac — Wed, 18 Feb 2009 02:26:23 GMT

QUESTIONS:
Is it important to configure the binding order on the "Adapters and Bindings" tab like in Microsoft Failover Clusters?

This is located in Network Connections from Control Panel, by going to the Advanced drop down menu then selecting Advanced Setting. Especially when you have Infiniband (IB) network which probably should appear last on the list.

So how can this order be controlled remotely from the head node (HN) and on each compute node (HN)? Any script to do this with clusrun?

ANSWER:

The order of the network binding is insignificant with CCS / MSMPI.
You should define the "Private" and MPI" networks from the TODO list on the HN.

Moreover, CCS sets the hosts files on each CN with the cluster node's NetBIOS name to Private network IP address mapping. So the default NetBIOS name of a CN is resolved to the Private network address.

What does matter, is the following:

If CCS is setup to only use Public for HN and CN, then go through the Manage Windows Firewall Setting (Wizard) under the Networking section of the To Do List and disable ICS/NAT.

If CCS is setup to only use Public/Private for HN and CN, then go through the Manage Windows Firewall Setting (Wizard) under the Networking section of the To Do List and enable ICS/NAT.

NOTE: When enabling ICS/NAT it act as a mini DHCP server to the CN, hence is the reason why a full blown DHCP Server does not need to be installed on the HN.

The MPI net mask is passed for every executed task as an environment variable CCP_MPI_NETMASK, which mpiexec picks and uses for mpi traffic.

For additional information regarding Microsoft Compute Cluster Server, please visit our Windows HPC (High Performance Computing) Community forums

Author: Mike Rosado
Senior Support Engineer
Microsoft Enterprise Platforms Support

Technorati Tags: CCS

CCS v1 - How does MPI network actually work without name resolution?

tomac — Wed, 18 Feb 2009 02:23:10 GMT

Technorati Tags: Compute Cluster Server,CCS

As you may already know, with the release of Windows Compute Cluster Server 2003 (CCS) we included Microsoft Message Passing Interface (MS‑MPI) implementation which is fully compatible with the reference MPICH2. This allows integration with Active Directory and enables role based security for administrators and users, and the use of Microsoft Management Console (MMC) which provides a familiar administrative and scheduling interface.

The Microsoft CCS can use GbE, InfiniBand (IB), Myrinet, Quadrics, or legacy high-speed fabrics as interconnects for high performance computing. The majority of high performance computing clustered systems use GbE, but more and more customers these days prefer the high speed and low latency of interconnects such as InfiniBand or legacy specialty hardware. Our implementation of CCS supports all WSD-compatible fabrics.

This is one of those things that you wake up some days wondering “How does this thing actually work?” Which seems to be a simple question, but then after couple of discussions with the developer you realize that “Hmm, actually it is not very clear or you say some magic is happening somewhere!” If you are trying to find out answers for the following questions, then listen up…

What magic happens during MPI initialization?
What are business cards, and how do MPI apps get these for other nodes?
How MPI network works without name resolution?

What’s more interesting that when we checked the test clusters with IB cards, we found that DNS and Default Gateway settings are not configured on IB network interface cards (NICs). There was no name resolution mechanism, on the MPI network at all. So how we force the MPI traffic using mpich subnet mask without name resolution…..

After thoroughly discussing this with Mr MPI , here is a brief summary on how magic happens…

Myth: Subnet manager running on IB switch does name resolution on MPI network?
Wrong !

User submits an mpi job

Job Scheduler allocates number of nodes (or processors) requested for mpi job.

First allocated node runs the mpiexec with all required parameters that are passed by job scheduler (ccp_nodes; ccp_mpi_network …etc)

mpiexec kicks off and forms a tree by talking first to the msmpi service running on the same node, which spawns the smpd manager talking to msmpi services running on other (allocated) compute nodes that’s where we need name resolution. Because smpd manager on the first node needs to talk to other msmpi service/smpds on allocated nodes

Each mpi application starts up and queries all the LOCAL addresses for that node. Then they register this information in a “business card” in a shared database inside the smpd tree business card has all available interfaces on the node.

When MPI app rank x running on node X needs to connect to MPI app rank y running on node Y, it get y’s bizcard from the smpd tree and connects directly to x, using the address list in the business card.

MPI app x filters y addresses using MPIHC_NETMASK environment variable. This environment variable is set by mpiexec by reading CCP_MPI_NETMASK; which in turn is set as a cluster variable by ccp management services. This var is set when you select the networks in the ToDoList; it set to the MPI network if selected or to the Private network if MPI network is not present.

So bottom line is we do not need to have a name resolution on MPI network as long as node can resolve their names through private or public network.

For additional information regarding Microsoft Compute Cluster Server, please visit our Windows HPC (High Performance Computing) Community forums

RELATED RESOURCE REFERENCE(S):

· Message Passing Interface (MPI) Documentation

· Using Microsoft Message Passing Interface (MS-MPI) Documentation

Mike Rosado
Senior Support Engineer
Microsoft Enterprise Platforms Support

Ask the Core Team : Cluster, CCS

CCS v1 - How to setup network adapters and bindings on Compute nodes ?

CCS v1 - How does MPI network actually work without name resolution?