The Windows HPC Team Blog

REST API docs on MSDN

Don Pattee — Wed, 25 Jan 2012 21:53:44 GMT

I know there were a bunch of folks waiting on this, so I'm happy to say the REST API documentation for the HPC Pack 2008 R2 SP3 release is available.

Windows HPC Server 2008 R2 with Service Pack 2 (SP2) provides access to the HPC Job Scheduler Service by using an HTTP web service that is based on the representational state transfer (REST) model. You can use this REST API to create client applications that users can use to define, submit, modify, list, view, requeue, and cancel jobs.

Windows HPC Server 2008 R2 with Service Pack 3(SP3) expands the HTTP web service to provide additional operations that provide information about nodes, node groups, and the cluster name. Windows HPC Server 2008 R2 with SP3 also provides operations that allow you to create and manage SOA sessions, and to send SOA requests and receive SOA responses for these sessions. The SOA operations that are available only when the REST web service is hosted in Windows Azure. All other operations are available either when the REST web service is hosted on an on-premise cluster or when the REST web service is hosted in Windows Azure.

Check it out at http://msdn.microsoft.com/en-us/library/hh560254(VS.85).aspx and http://msdn.microsoft.com/en-us/library/hh560258(VS.85).aspx

Interoping Windows HPC SOA from Java (or other non-.NET environment)

yidingz — Sat, 31 Dec 2011 03:19:00 GMT

When working with our customers to adopt Windows HPC cluster, some of them asked for interop capability so that they can integrate legacy Linux or Java based application into Windows HPC infrastructure. After a major release (Windows HPC Server 2008 R2) and 2 service packs (SP1 & SP2), I'm happy to introduce you the Java interop library for Windows HPC SOA.

Quick Start In 30 Seconds

If you know what you need, just download https://github.com/MicrosoftHPC/Java-Interop-Library/tarball/master, unpack it, and follow the README.

So What Is It?

"Java Interop Library For Windows HPC Server" is an open-sourced, production quality, redistributable, and commercial software friendly library, which is also fully supported by Microsoft HPC team. The library enables a Java client application to allocate resource and run parallel computing code on Windows HPC cluster. This library also include a Java based "service host" that hosts Java computation code to run HPC cluster.

The library is open sourced (see below) and the source code can be downloaded on www.github.com. The library is distributed under a Apache-like license so you can modify, compile and redistribute it in both binary and source code format provided it's operated with Windows HPC Server. It also means you can do it internally, for free or make money from it (as part of some commercial software).

The library is backed up by Microsoft HPC team and has been fully tested by the team. HPC team commit full support on the library. This means that we will listen to the feedback from our users, fix bugs, and potentially add features to align with the .Net library.

The client library talks through SOAP over HTTP protocol with our Windows HPC cluster. HPC SOA is based on WCF and provide service on both SOAP/HTTP and Net.TCP binding end points. All those acronyon just means that the all functionalities on HPC cluster can be access from SOAP/HTTP client (i.e., the Java client) as well as Net.TCP based client (i.e., the .Net/C# client that comes with HPC Client Pack). In fact, the Java client does implement almost all C#'s functionalities.

Here is a quickly list of things the Java library can do,

Works on both Windows and Linux. (We have tested on RHEL 5.5)
Create a session (Durable & Non-Durable) on HPC cluster, and control all Session related properties.
Send batch of requests to the session.
Async send/receive pattern.
Flush/Cancel/Cleanup of request batch.
Using the common data API to stage large data in and out of cluster efficiently.
All scheduling policies work. That means first come first serve, priority, preemption, dynamic grow/shrink of allocated resource, graceful killing, automatic error retry, and high availablity.

Besides the client library, it also comes with a Java based service host, means that user's Java based calcluation engine can be hosted on Windows HPC OS natively without a C++ or C# wrapper.

So It's Working, How's the performance?

The real throughput number depends on a lot of factors - message size, network connection, broker node CPU/memory configuration, and number of clients/services running. Following is the measured end-to-end message throughput of a reasonabled configured cluster. (Both the broker and the client were running AMD Opteron 1.9GHz with 24 cores and 32GB of memory. Both clients and services are connected to the broker with 1GB Ethernet.)

Metrics	0B	1KB	2KB	4KB	8KB	16KB	32KB	64KB	128KB
Interactive Session - 16 cores	593	542	556	572	524	522	447	338	211
Interactive Session - 256 cores	1867	1699	1610	1694	1443	1227	855	457	248
Durable Session - 16 cores	422	422	443	427	405	368	313	254	166
Durable Session -256 cores	1526	1515	1269	1119	895	448	253	128	80

As a reference point, the throughput of Durable Session are roughly the same as Durable Session throughput with C#/.Net client. For a Durable Session, since all messages are persisted onto disk, the bottleneck is the CPU and disk I/O so there is no significant performance difference between Java and C# client. The interactive session, however, is about 50% slower than C#/.Net client - a C# client will have throughput around 5000msg/sec. The main reason is that for an Interactive Session, the bottleneck is the network since there is no disk I/O involved. Dot Net client use Net.tcp binding which use binary encoding and compression so is faster than the http binding used by Java client. This is the price we pay for interopbility.

OK, Great. Now Where Do I Get It?

Interested? Good. Because it's easy to get! Just open https://github.com/MicrosoftHPC/Java-Interop-Library, go to the download section, and choose the package right for you.

Or, if you prefer git, checkout a local version as you want. The website has full instruction on how to do this. Please notice that at this stage, we will *NOT* accept any change from the community.

I Got It. I Want To Use It! Right Now!

The README in the package comes with instruction on how to use the library. The "test/" directory comes with the test suite we use to validate the library. And you know "sample/helloworld" is always a good idea to start with everything.

Following is an excerpt from the hello world client program with comments. The code is simple enough to be self-explanatory.

 01 // step 1. Create a session
 02 SessionStartInfo info = new SessionStartInfo(headnode, serviceName, username, password);    
 03 DurableSession session = DurableSession.createSession(info);
 04 
 05 System.out.printf("new session id = %d\n", session.getId());
 06 
 07 // step 2. Create a client
 08 BrokerClient<CcpEchoSvc> client = new BrokerClient<CcpEchoSvc>(session, CcpEchoSvc.class);
 09 for(int i = 0; i < 10; i++)
 10 {
 11   // step 3. Create the message and send it to broker
 12   ObjectFactory of = new ObjectFactory();
 13   Echo request = of.createEcho();
 14   request.setInput(of.createEchoInput("hello world!"));
 15   client.sendRequest(request, i);
 16 }
 17 // step 4. endRequests() must be called to start processing.
 18 client.endRequests();       
 19 
 20 // step 5. the foreach() call will iterate through all the responses
 21 for(BrokerResponse<EchoResponse> response : client.<EchoResponse>getResponses(EchoResponse.class))
 22 {
 23   try
 24   {
 25     String reply = response.getResult().getEchoResult().getValue();
 26     System.out.printf("\tReceived response for request %s: %s%n", response.getUserData(), reply);
 27   }
 28   // if the message hasn't been processed correctly, an Exception will be generated.
 29   // However, other messages might have been processed, so should continue process.
 30   catch(Exception ex)
 31   {
 32     nerrs++;
 33     System.out.printf("Error: process %s-th reuqest: %s%n", response.getUserData(), ex.toString());
 34   }
 35 }
 36 
 37 // step 6. remember close the client and session
 38 client.close();
 39 session.close();

Above is how you consume a service already deployed on Windows HPC cluster - regardless the service is based on .Net or Java. But, how to host a service created in Java? It turns out to be pretty easy too. Basically there are 2 things you need to accomplish. First, you need modify the service registration files to indicate it's a Java based service. Second, you need use the Java service host library to create the Java service host.

Following is a segment of the JavaEchoSvc.config since the rest of configuration are the same as service config for a .Net based service. Two things to note here. First, the service assembly field is changed to pointing to a JAR file instead of .Net assembly. This is the actual Java service that'll be loaded. In this case, we assume it's been put under C:\JavaSvcHostTest\. Second, the service host is customized to replace the default .Net service host. The Microsoft-HpcServiceHost-3.0.jar should be compiled with the source code you get from the github project. There are no other changes to the service registration.

 <microsoft.Hpc.Session.ServiceRegistration>
 <service assembly="C:\JavaSvcHostTest\JavaEchoSvc.jar" ... > ... </service>
 <!-- Using Java Service Host -->
 <host hostType="Customize" exeFileName="java -jar &quot;%CCP_HOME%bin\Microsoft-HpcServiceHost-3.0.jar&quot;" />
 </microsoft.Hpc.Session.ServiceRegistration>

The second part is to create the actual Java service host. It's not that complicated either, as you can see from the CcpEchoSvc under "sample/" directory.

 01 package org.tempuri;
 02 
 03 import javax.xml.bind.JAXBElement;
 04 import javax.xml.bind.annotation.XmlAccessType;
 05 import javax.xml.bind.annotation.XmlAccessorType;
 06 import javax.xml.bind.annotation.XmlElementRef;
 07 import javax.xml.bind.annotation.XmlRootElement;
 08 import javax.xml.bind.annotation.XmlType;
 09 
 10 
 11 /**
 12  * <p>Java class for anonymous complex type.
 13  * 
 14  * <p>The following schema fragment specifies the expected content contained within this class.
 15  * 
 16  * <pre>
 17  * &lt;complexType>
 18  *   &lt;complexContent>
 19  *     &lt;restriction base="{http://www.w3.org/2001/XMLSchema}anyType">
 20  *       &lt;sequence>
 21  *         &lt;element name="input" type="{http://www.w3.org/2001/XMLSchema}string" minOccurs="0"/>
 22  *       &lt;/sequence>
 23  *     &lt;/restriction>
 24  *   &lt;/complexContent>
 25  * &lt;/complexType>
 26  * </pre>
 27  * 
 28  * 
 29  */
 30 @XmlAccessorType(XmlAccessType.FIELD)
 31 @XmlType(name = "", propOrder = {
 32     "input"
 33 })
 34 @XmlRootElement(name = "Echo")
 35 public class Echo {
 36 
 37     @XmlElementRef(name = "input", namespace = "http://tempuri.org/", type = JAXBElement.class, required = false)
 38     protected JAXBElement<String> input;
 39 
 40     /**
 41      * Gets the value of the input property.
 42      * 
 43      * @return
 44      *     possible object is
 45      *     {@link JAXBElement }{@code <}{@link String }{@code >}
 46      *     
 47      */
 48     public JAXBElement<String> getInput() {
 49         return input;
 50     }
 51 
 52     /**
 53      * Sets the value of the input property.
 54      * 
 55      * @param value
 56      *     allowed object is
 57      *     {@link JAXBElement }{@code <}{@link String }{@code >}
 58      *     
 59      */
 60     public void setInput(JAXBElement<String> value) {
 61         this.input = ((JAXBElement<String> ) value);
 62     }
 63 
 64 }

Looks Simple Enough! What Can I Expect In The Future?

This is not the end of the library. There are still plenty of things we are planning. Besides adding support of new functionality that we will add in future release of Windows HPC SOA, we are also looking at the support of the new RESTful API and integration with Windows Azure HPC Scheduler (WAHS). In fact, there is a open source project (https://github.com/MicrosoftHPC/REST-Client-Sample) today that implement a Java library which talks with HPC cluster in REST protocol. Note that this project also comes with a C++ implementation too. This project is developed by Microsoft HPC team too. However, it's purposed as sample only so the quality is as-is and should not be granted as production level code. We'll keep working on the RESTful API support of Java and will probably release that with Java interop library, depending on customer requirements. When that library is released, it'll also be able to communicate with WAHS so the Java client will have a taste of the cloud.

Here is Java, What About Other Languages?

For now, the library is based on Java because this is what our customer has asked the most. However, we understand that there are a large portion of our customer still work on non-managed code. So a C/C++ based library is also in the pipeline.

So far, I've covered the basic features, performance and roadmap of the Java interop library. If you want to try it out, you know where to get it. And there is more to explore in the package. If you still have questions, the Windows HPC Server Developers forum is always a good place to post them.

Announcing the Windows Azure HPC Scheduler and HPC Pack 2008 R2 Service Pack 3 releases!

Don Pattee — Fri, 11 Nov 2011 23:00:00 GMT

Once again I get the honor of announcing, on behalf of the Microsoft High Performance Computing team, our latest releases are available immediately!

Our new Windows Azure HPC Scheduler development kit, and
the third update to the HPC Pack 2008 R2 family of software

Windows Azure HPC Scheduler

This is a great new product for us, the preview version was previously announced at the BUILD conference, and now we have officially released it for application developers to start using right now!

Windows Azure HPC Scheduler SDK includes modules and features that developers can use to create Windows Azure deployments that support compute-intensive, parallel applications that can scale when offered more compute power. The Windows Azure HPC Scheduler SDK enables developers to define a Windows Azure deployment that includes built-in job scheduling and resource management, runtime support for MPI, SOA, web-based job submission interfaces, and persistent state management of job queue and resource configuration. So, basically you can create an application that builds an app-specific cluster with no on-premises cluster requirements - this is cool :) Of course, applications that have been built using the HPC Pack's on-premises job submission API can use very similar job submission interfaces in the Windows Azure HPC Scheduler.

Get more details, and links to docs and sample code, from our Windows Azure HPC Scheduler MSDN page.

The Windows Azure HPC Scheduler SDK works with the newest version of the Windows Azure SDK (November 2011) which includes its own cool set of features to help develop Windows Azure applications, available for installation through the Web Platform installer here.

HPC Pack 2008 R2 Service Pack 3

This update includes a number of improvements, including two frequently requested features:

The Windows Azure bursting scenarios has reduced the number of ports you have to open in your firewall, using port 443 for most communication
The ability to install the HPC Pack software on a server not dedicated to your cluster (e.g. a team file server) for use in a manner similar to the existing Workstation Node functionality

The new 'Cycle Harvesting' feature is available to anyone who has a license for the Workstation or Enterprise versions - you'll need to download the update from your VL download page or by using the SP3 Integration Pack + your original Workstation or Enterprise media to create the new installer.

As part of this release we’ve also updated the preview version of LINQ to HPC, however, this will be the final preview and we do not plan to move forward with a production release. In line with our announcement in October at the PASS conference we will focus our effort on bringing Apache Hadoop to both Windows Server and Windows Azure. Hadoop has emerged as a great platform for analyzing unstructured data or large volumes of data at low cost, which aligns well with Microsoft’s vision for its Information Platform. It also has a vibrant community of users and developers eager to innovate on this platform. Microsoft is keen to not only contribute to this vibrant community, but also help its adoption in the Enterprise. We expect a preview version on Windows Azure available by end of the calendar year.

For more information on those, and other, new features available in Service Pack 3 please see our documentation on TechNet.

Note: The single SP3 installer applies to all installations - Express, Workstation, and Enterprise, as well as the standalone 'Client Utilities' and 'MS-MPI' packages. You can download it from the Microsoft Download Center. Installers for the standalone Client Utilities and MS-MPI packages with the service pack already integrated are also available.

If you do not have an HPC Pack 2008 R2 cluster, you can download a free Windows HPC Server 2008 R2 Suite evaluation version. Before you install, you can try out the new Installation Preparation Wizard which can help analyze your environment for common issues and provide some best practice guidance to help ensure an easy HPC cluster setup.

Head over to the Windows HPC Discussion forums if you have any questions or comments, we'll be happy to hear from you!

Preview of Windows Azure Scheduler and the HPC Pack 2008 R2 Service Pack 3 releases now available

Don Pattee — Tue, 18 Oct 2011 00:04:00 GMT

Microsoft's High Performance Computing team has just made our 'release candidate' for two products available: The HPC Pack 2008 R2 Service Pack 3 and the Windows Azure Scheduler SDK.

The HPC Pack service pack is an update to the same Windows HPC cluster software that you know and love, with improvements to basic functionality & stability and a few additional new features such as the integration of the Linq to HPC runtime (previously released as a beta add-on, check out the BUILD conference presentation at http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-453T for more information), enhancements to our Windows Azure bursting scenarios by reducing the number of ports you have to open in your firewall (services now use 443 instead of a multiple ones), and the ability to install the HPC Pack software on a server not dedicated to your cluster (e.g. a team file server) for use in a manner similar to the Workstation Node functionality previously available.

The big new part is the first chance at trying out the Windows Azure Scheduler, previously announced at the BUILD conference (http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-452T)

The Windows Azure Scheduler for Parallel Applications is a solution that enables you to deploy applications in a scalable, high-performance computing (HPC) infrastructure in Windows Azure. With the Windows Azure Scheduler, you can schedule, submit, and monitor HPC jobs that use your Message Passing Interface (MPI), service-oriented architecture (SOA), or LINQ to HPC applications.

With the Windows Azure Scheduler SDK, you can create Windows Azure deployments that support scalable, compute-intensive, parallel applications. This SDK provides the following features:

Built-in job scheduling and resource management.
Runtime support for Message Passing Interface (MPI).
Service-oriented architecture (SOA).
LINQ to high performance computing (HPC) applications, web-based job submission interfaces, and persistent state management of job queue and resource configuration.

Applications that have been built using the on-premises job submission API in Windows HPC Server 2008 R2 can use very similar job submission interfaces in the Windows Azure Scheduler.

To get access to these pre-release installers head over to the HPC team's "Connect" beta website ( http://connect.microsoft.com/HPC ). Once you sign up, you'll have access to the Release Candidate on the website's "downloads" section.

For questions and comments head over to our discussion forums at http://social.microsoft.com/Forums/en-US/category/windowshpc

HPC presentation from BUILD

Don Pattee — Tue, 20 Sep 2011 20:06:00 GMT

Come watch our development manager, Greg Burgess, give a talk on how HPC applications can be made to work in Windows Azure.

Windows Azure is an ideal environment for deploying compute-intensive apps that take advantage of the scale-on-demand capability of the cloud. The HPC Pack for Windows Azure provides a job scheduler, management tools, and a set of runtimes for developing and deploying parallel and scale-out apps. This talk will present app examples and best practices for using the HPC Pack in Windows Azure to rapidly create scalable compute and data-intensive services. Programming models include parallel apps using MPI, scale-out apps using WCF and data intensive apps using HPC to LINQ.

http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-452T

If the talk and the idea of making an application that can easily burst to the cloud or bulding a Windows Azure-based HPC server excites you, sign up with our beta program so you can be one of the first to try out the new tools when they become available: http://connect.microsoft.com/HPC/content/content.aspx?ContentID=6923

Database capacity planning

Don Pattee — Thu, 01 Sep 2011 00:15:12 GMT

We've just published new guidance on how to do capacity planning and tuning for the database that runs your HPC Pack 2008 R2-based cluster.

Visit Database Capacity Planning and Tuning in Windows HPC Server 2008 R2 and then let us know what you think over on our Windows HPC Administration Forum.

Enabling tracing for HPC SOA applications

Don Pattee — Thu, 28 Jul 2011 21:35:16 GMT

When helping our customers troubleshoot HPC SOA applications and cluster infrastructure, we realized that SOA trace is a very powerful tool when physical or remote logon is not available to the cluster. (Which happens a lot given that HPC is always a core asset of institutes and normally hosts sensitive or private data.)

However, we found that there is no clear and central instruction on how to enable HPC traces! So here is wrap-up of (hopefully) everything.

First of all, there are 3 levels of traces, let's take a look of them one by one.

Closest to application level, there is the user code trace, which is generated by user service code but service host also generates information about receiving and sending message. This is helpful in troubleshooting errors in user code and monitoring message traffic on compute nodes. The easiest way to enable this trace is through HPC Cluster Manager - launch HPC Cluster Manager, switch to the Configuration pane, select Service node, select the service you want to configure trace, click "Configure Trace" in the action pane on the right (or through right-click menu), and select the desired trace level. I'll suggest using Verbose’ directly because there is no such thing as information overload.

NOTE: The configuration will only take effect on those processes that start AFTER the modification. So make sure you modify the setting before you run the session.

Viewing this trace is also easy - launch HPC Cluster Manager, switch to the Job pane, select the job you want to view the trace of, click "Collect trace" in the action pane on the right (or through right-click menu), and replace the directory in the dialog if necessary. Note that this path must be accessible from all compute nodes, so the file share on head node is a good idea.

After the collection finishes, you will see all the svclog files have been put in the specified directory. To view those files, use the Service Trace Viewer available with the Windows SDK or Visual Studio 2008 (or later).

The 2nd type of trace is SOA infrastructure trace. This trace provides more "insight" on what's happening under the hood of HPC SOA Session service, Broker service, broker worker, and service host processes. This is the main source of information when you hit problem that you can't solve and have to turn to Microsoft for help. To enable the trace, on the headnodebroker node, launch an admin privilege CLI. Run,

logman start trace SOATRACE -p Microsoft-HPC-Runtime -max 2000 -o "%CCP_DATA%\LogFiles\SOATrace.etl" -ets

Run the session again, and SOATrace.etl will be generated under %CCP_DATA%\LogFiles. To view an .etl file, you can run “Event Viewer” and load the file by menu item “Action -> Open Saved Log…”.

NOTE: Remember to stop the trace once you finished troubleshooting since trace has a negative impact on broker performance.

The last type of trace is WCF tracing. In case nobody understands what the problem is, WCF library can log detailed information at message level and might be helpful. To enable WCF traces, replace the <system.diagnostics> section of %CCP_HOME%\bin\HpcBrokerWorker.exe.config on all broker nodes,

<system.diagnostics>
    <sources>
      <source name="System.ServiceModel" switchValue="Verbose,ActivityTracing"
        propagateActivity="true">
        <listeners>
          <add type="System.Diagnostics.DefaultTraceListener" name="Default">
            <filter type="" />
          </add>
          <add name="WcfTraceListener">
            <filter type="" />
          </add>
        </listeners>
      </source>
      <source name="System.ServiceModel.MessageLogging" switchValue="Verbose,ActivityTracing">
        <listeners>
          <add type="System.Diagnostics.DefaultTraceListener" name="Default">
            <filter type="" />
          </add>
          <add name="WcfTraceListener">
            <filter type="" />
          </add>
        </listeners>
      </source>
    </sources>
    <sharedListeners>
      <add initializeData="c:\temp\hpcbrokerworker.trace.svclog" type="System.Diagnostics.XmlWriterTraceListener, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
        name="WcfTraceListener" traceOutputOptions="LogicalOperationStack, DateTime, Timestamp, ProcessId, ThreadId, Callstack">
        <filter type="" />
      </add>
    </sharedListeners>
</system.diagnostics>
nbsp;

NOTE: Just like user code trace, changing app.config file requires a restart of the executable. So make sure you modify the setting before you run the session. You also need restart the HPC Broker service on broker nodes after making this change.

NOTE: Make sure the user has ‘write’ privilege to the specified directory, in this case “c:\temp”.

NOTE: This is for SOA broker worker. To understand the end-to-end workflow, make sure you modify %CCP_HOME%\bin\HpcServiceHost.exe.config on all compute nodes and the app.config of you client application on client computer.

A pair of new release updates

Don Pattee — Fri, 22 Jul 2011 21:48:25 GMT

We have released a new set of 'sample code' that uses some of the new features available in the HPC Pack 2008 R2 Service Pack 2 release. You can download it from the SP2 SDK page: http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=26645

We've also updated the Tool Pack to include a 'Ganglia Integration' feature. If you use the Ganglia monitoring system, this update will let you monitor your HPC Pack 2008 R2 SP2-based cluster. Check it out at http://www.microsoft.com/download/en/details.aspx?id=8433

As usual, if you have comments on either of those items or suggestions for future additions please head over to the Windows HPC forums: http://social.microsoft.com/Forums/en-US/category/windowshpc

Deep Thread: Tracing HPC Deployments on Windows Azure

Don Pattee — Mon, 18 Jul 2011 21:05:00 GMT

One of the things we’ve worked hard to do here in the Windows HPC team is make it easy for the HPC cluster admin to deploy and manage Windows Azure nodes. By and large, admins that are familiar with the HPC paradigms and processes should be comfortable deploying and managing their Windows Azure nodes.

Most of the time things work well, however there can be occasions where the cluster admin has to troubleshoot and analyze where deployments don’t quite work as expected. The Diagnostics suite that ships with Windows HPC Pack 2008 R2 (and which has been beefed up in SP2 for Windows Azure deployments) should be the first point of troubleshooting.

For the admin who wants to dig a bit deeper, or is working in tandem with Microsoft Support to diagnose issues, we ship a command-line tool with Windows HPC Pack 2008 R2 that allows you to operate under the covers of what is happening inside of their HPC deployment on Windows Azure. This tool is hpcazurecmd, and below are some details of how to use it to get traces from Windows Azure.

Warning: The hpcazurecmd tool is a powerful tool, intended for advanced users. Usage without knowing what you are doing could have un-intended consequences for your Azure deployments. Please exercise appropriate care when using this tool.

One of the coolest things about Windows Azure is the powerful and dynamic tracing toolset that it provides. For those not familiar with it, you can see some details here: http://msdn.microsoft.com/en-us/magazine/ff714589.aspx

Windows HPC takes full advantage of this dynamic tracing. By default, Windows Azure tracing is turned off. It can be turned on, on-the-fly, with hpcazurecmd. Turning on the tracing copies over traces to a table in the Windows Azure storage that was previously configured for use with a HPC Node Template, for a given deployment. And hpcazurecmd also provides a way to retrieve the traces from the table right to the desktop, so the entire process of turning on and retrieving traces becomes very streamlined.

The steps below assume you have already created a Node Template for your Windows Azure deployment (let’s say, named ‘hpcazuretemplate’), have added some nodes based on this template, and started the deployment.

Start by opening up an elevated command prompt on the Head Node, navigating to the HPC bin folder, and setting an environment variable to point to the template name.

C:\Program Files\Microsoft HPC Pack 2008 R2\Bin>set TemplateName=hpcazuretemplate

Now, issue a command to get the deployment information:

C:\Program Files\Microsoft HPC Pack 2008 R2\Bin>hpcazurecmd /GetDeployments

This should give some output such as the below. Note the Name field, as you will need to copy and paste it into the next command as a parameter.

Get Windows Azure configuration information from template "hpcazuretemplate"
1 Windows Azure deployment found
Deployment: 4865f261-af99-4bc5-9a68-9320d1234567
Name: hpcdeploymentmachinenamehnservicename75c5e023db8346758531fd6123456
Url:
label: RGVwbG95bWVudCBmb3IgTWljcm9zb2Z0IFdpbmRvd3MgSFBDIDIwMDggUjIgQ2x1c3Rlcjogc
2FsaW1hLWRldi1obg==
Status: Deploying

If the deployment has not yet started, you will see something like:

Get Windows Azure configuration information from template "hpcazuretemplate"
0 Windows Azure deployments found

Wait until the deployment has started, and you can see the deployment information as above. Now issue the command to set the logging level that you desire – e.g.:

C:\Program Files\Microsoft HPC Pack 2008 R2\Bin>hpcazurecmd /settracinglevel /name: hpcdeploymentmachinenamehnservicename75c5e023db8346758531fd6123456 /Level:verbose

Note that the ‘name’ parameter in the command should be the same as the ‘Name’ from the output of the ‘hpcazurecmd /getdeployments’ command – this is what identifies the specific deployment you will get the traces for. You can copy and paste it from within the command line window. The trace level here is set to Verbose, which produces the most detailed trace data.

Next, you need to wait for your deployment to move through the provisioning stage. This can generally take a few minutes. You can then type in the following command to retrieve the current trace log (with the correct name, of course):

C:\Program Files\Microsoft HPC Pack 2008 R2\Bin>hpcazurecmd /gettracinglog /name: hpcdeploymentmachinenamehnservicename75c5e023db8346758531fd6123456 /output:log.txt

The output will tell you the number of trace entries that were retrieved, e.g.:

Get Windows Azure configuration information from template "hpcazuretemplate"
Deployment Id is 4865f261af994bc59a689320d1234567
....................................
Total 36 entries

Now you can open up ‘log.txt’ in your favorite editor and examine the contents of the trace.

Note that if you get the following error output, you need to wait for the deployment to move beyond the initial provisioning stage, as the trace table is not yet available. You can check the Node State in HPC Cluster Manager – if the nodes are still in the “Provisioning” stage, they may not have yet had any opportunity to log anything.

Get Windows Azure configuration information from template "hpcazuretemplate"
Deployment Id is 4865f261af994bc59a689320d1234567
System.Data.Services.Client.DataServiceQueryException: An error occurred while p
rocessing this request. ---> System.Data.Services.Client.DataServiceClientException:
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
  <code>TableNotFound</code>
  <message xml:lang="en-US">The table specified does not exist.
RequestId:1a158cad-0441-446e-95dd-8a5421234567
Time:2011-07-14T19:48:02.0425613Z</message>
</error>
   at System.Data.Services.Client.QueryAsyncResult.Execute(MemoryStream requestContent)
   at System.Data.Services.Client.DataServiceRequest.Execute[TElement](DataServiceContext context, Uri requestUri)
   --- End of inner exception stack trace ---
   at System.Data.Services.Client.DataServiceRequest.Execute[TElement](DataServiceContext context, Uri requestUri)
   at System.Data.Services.Client.DataServiceQuery`1.Execute()
   at System.Data.Services.Client.DataServiceQuery`1.GetEnumerator()
   at Microsoft.Hpc.AzureCmd.GetTracingLog.RunCommand(IDictionary`2 propertyMap, IAzureManagementBroker broker)
   at Microsoft.Hpc.AzureCmd.HpcAzureCmd.Main(String[] args)

A note on trace levels: there are a number of trace levels that can be specified, such as Error, Warning, etc. (you can get the full list and usage info by just typing ‘hpcazurecmd’). Generally, the trace output will get larger as you go from Error on up. Verbose traces can spew very large amounts of data, so be extremely careful when turning on Verbose, especially for deployments with large sets of nodes as they can generate multi-gigabytes worth of data in your Azure storage (you may want to delete the WADLogsTable from your Azure storage once you are done with your deployment).

Hopefully this post helped you gain more understanding of Windows HPC and Windows Azure under the covers. Good luck in your investigative journeys!

Salim Alam, Windows HPC Development Lead

Announcing LINQ to HPC Beta 2

Don Pattee — Thu, 07 Jul 2011 20:45:00 GMT

We’re pleased to announce the availability of LINQ to HPC Beta 2.

LINQ to HPC enables a new class of data intensive applications for Windows HPC Server by providing a sophisticated distributed runtime and associated programming model for large scale, unstructured data analysis that is easy to use and program. For deeper insights, it integrates with SQL Server 2008, SQL Azure, and the rich portfolio of Business Intelligence offerings from Microsoft such as SQL Server Reporting Services, SQL Server Analysis Services, PowerPivot, and Excel. IT Professionals and developers now have a unified HPC platform that can run both compute and data intensive HPC applications.

Key features of LINQ to HPC include:

LINQ to HPC enables a new class of data intensive applications by providing a sophisticated distributed runtime and associated programming model for analyzing large volumes of unstructured data using commodity clusters. LINQ to HPC is offered as an integral feature of Windows HPC Server, providing a unified scale-out platform that can run both compute and data intensive HPC applications.
LINQ to HPC is based on LINQ, a powerful, language integrated query model, offering a higher level of abstraction that makes it easy to develop distributed, scale-out data intensive applications. It integrates with Visual Studio 2010 and the .NET framework for a superior development experience and increased developer productivity.
For deeper insights, LINQ to HPC can integrate with SQL Server 2008, SQL Azure and the rich portfolio of Business Intelligence offerings from Microsoft such as SQL Server Reporting Services, SQL Server Analysis Services, PowerPivot, and Excel.
Through Windows HPC Server 2008 R2, LINQ to HPC offers a familiar, easy to use and easy to manage environment that lets you take advantage of your existing IT infrastructure, thereby reducing the learning curve and increasing return on investment.

For more information about LINQ to HPC see Introducing LINQ to HPC and the documentation accompanying the Beta 2 download.

The Beta 2 installer, along with documentation and samples, can be downloaded from https://connect.microsoft.com/hpc. LINQ to HPC Beta 2 requires HPC Pack 2008 R2 SP2.

HPC Pack 2008 R2 Service Pack 2 (SP2) is now available!

Don Pattee — Wed, 29 Jun 2011 17:38:00 GMT

The second service pack to the HPC Pack 2008 R2 software is now available!

This update includes a number of great new features, including

Enhanced Azure capabilities, such as adding Azure VM nodes to your cluster, creating Azure node configuration scripts, and supporting Remote Desktop connections
Ability to run MPI-based applications in Azure
Job scheduling support through a REST interface and an IIS-hosted web portal
A new job scheduling policy that uses 'resource pools' to ensure compute access to different user groups
SOA improvements, such as in-process broker support for increased speed, and a new common data staging feature
and all the 'normal' service pack stability improvements

For more information on those, and other, new features available in Service Pack 2 please see our documentation on TechNet.

The single SP2 installer applies to both Express and Enterprise installations, as well as the standalone 'Client Utilites' and 'MS-MPI' packages. You can download it from the Microsoft Download Center.

An important note is that this service pack can not be uninstalled by itself. Uninstalling the service pack will also uninstall the HPC Pack itself, so you'll need to take a full backup (including the sql databases) before installation if you want to be able to 'roll back.'

If you do not have an HPC Pack 2008 R2 cluster, you can download a free Windows HPC Server 2008 R2 evaluation version. Before you install, you can try out the new Installation Preparation Wizard which can help analyze your environment for common issues and provide some best practice guidance to help ensure an easy HPC cluster setup.

Let us know your thoughts over on our Windows HPC discussion forums.

C++ Accelerated Massive Parallelism

Don Pattee — Thu, 23 Jun 2011 18:00:00 GMT

One of the HPC team’s “sister groups” is the Parallel Computing Platform team, which creates tools that allow developers to efficiently harness the power of multicore processors. We are pleased to say that they have recently announced a new technology called “C++ Accelerated Massive Parallelism” (with the sweet acronym “C++ AMP”) that will ship with the next version of Visual Studio.

This new technology will help C++ developers easily use GPUs from any vendor in their code!

Please check out their official announcement on Soma's Blog and follow Daniel Moth's blog for more information.

Uninstalling HPC Pack 2008 R2

Don Pattee — Wed, 22 Jun 2011 20:56:00 GMT

If for some reason you want to uninstall the HPC Pack 2008 R2 software from multiple computers, read on...

HPC Pack 2008 R2 is made up of several different components (for example the Client Utilities and the MPI Redistributable) that get installed in different combinations depending on what options you picked during setup. To fully remove the HPC Pack software you must remove all of the components, and some of the components can only be removed in a specific order.

It is not difficult to use the" Add or Remove Programs" or "Programs and Features" control panel to uninstall the pieces from 1 or 2 computers, but if you have a lot of computers to update that can get very tedious.

Copy this script in to notepad and save it as 'UninstallHPC.bat', and you can use it on any computer to remove all HPC Pack 2008 R2, HPC Pack 2008, and Compute Cluster Pack software.

echo Removing Compute Cluster Pack

MsiExec.exe /uninstall {01493E6E-2473-4DE5-963B-BF17BACC21C3} /passive
MsiExec.exe /uninstall {C87F3322-A8A0-4239-A81C-36292A250593} /passive

echo Removing HPC Pack 2008

MsiExec.exe /uninstall {21CE1D35-AB84-46fe-B2FD-B57271BE7B93} /passive
MsiExec.exe /uninstall {3EEC9986-EDE0-4ee2-ACFB-A3CD78A16F6D} /passive

echo Removing HPC Pack 2008 R2

echo Removing HPC Server Components
MsiExec.exe /x{CD5190DD-A85D-4844-9BF4-AC6B04EB1A12} /passive
MsiExec.exe /x{F1BB42C1-B9AE-4F47-843A-12FE8F956FFA} /passive
echo Removing MPI Redistributable
MsiExec.exe /x{D3299935-57F7-403A-9D7B-0B8F9F56F44B} /passive
MsiExec.exe /x{5576D439-C958-4EB6-97D9-EE9EA5760DAF} /passive
echo Removing Client Utilities
MsiExec.exe /x{D86BF5A7-BB6E-423f-AA1D-02B5F59C38B0} /passive
MsiExec.exe /x{67E5BC0C-8BFE-450C-902E-FDE8660E02E1} /passive
echo Removing HPC Excel Components
MsiExec.exe /x{7544D810-316A-4380-BBD6-3B7C02962392} /passive
MsiExec.exe /x{14AFEF5F-DD16-48FF-BFF9-2B4A83352422} /passive
echo Removing HPC Web Components
MsiExec.exe /x{7160D5B0-9F57-4B72-B4CA-41F65ABF44B7} /passive

Notes:

This does not remove your database files or any WIMs that you've captured, so you may still have 'HPC-related files' on your computer.
The script needs to be run from an Elevated/Administrator command window or you will have to acknowledge pop-up windows for each step.

Creating a 'Service Pack integrated' installation point

Don Pattee — Thu, 09 Jun 2011 19:01:00 GMT

If you are only deploying a few new clusters across your network, then applying the latest HPC Pack service pack to each head node prior to deploying its compute nodes is a pretty simple workflow. You install your head node in your favorite way (either by walking through the setup wizard, or by using the unattended installation command line options to setup.exe), then you install the service pack (interactively or via unattended command line installation), reboot, and you are ready to rock and roll - any compute node that you deploy from that head node will automatically have the service pack version of the files installed.

But what if you are deploying a dozen different clusters? Then the extra 5 minutes per head node to install the service pack & reboot starts to add up!

In that case you can follow this very simple procedure to create an installation folder on your network that can be used to create a new head node with the service pack already 'built-in.'

Copy the entire contents of your original HPC Pack 2008 R2 DVD to a folder on your computer i.e. c:\HPCPackInstall (This will take approximately 1.5 gigabytes of space)
Download the HPC Pack 2008 R2 Service Pack 2 'media integration pack' zip file from the Microsoft Download Center
Extract the files from the zip archive to a folder on your computer i.e. c:\HPCSP2Files
Open up the folders you created in step 1 and step 3 (i.e. c:\HPCPackInstall, c:\HPCSP2Files) in Explorer
Drag all the files from the second (c:\HPCSP2Files) folder in to the c:\HPCPackInstall folder, and for any 'Copy File' dialogs that come up select "Copy and Replace" and in any 'Confirm Folder Replace' dialogs select 'Yes' to 'Merge' the folders.

The new installation folder that you created is now ready to use. Any head node that you install from this new installation point will automatically have the latest service pack files installed, and any compute nodes deployed from that head node will of course also have the latest service pack files installed automatically.

If you want to make an integrated Service Pack 1 installation point follow these steps: (Since SP2 has already been released, you should follow the instructions above so that you have the latest supported software!)

Copy the entire contents of your original HPC Pack 2008 R2 DVD to a folder on your computer i.e. c:\HPCPackInstall (This will take approximately 1.5 gigabytes of space)
Download the HPC Pack 2008 R2 Service Pack 1 installer from the Microsoft Download Center
If you do not currently have an HPC Pack 2008 R2 head node set up, install it now
Install the HPC Pack 2008 R2 Service Pack 1 on the head node, follow all the standard Service Pack installation steps, rebooting as necessary.
Open up the folder you created in step 1 (i.e. c:\HPCPackInstall) in Explorer
Open up the folder that the head node uses to deploy new nodes in another Explorer window - this is the special REMINST file share on your headnode, so \\<headnode-name>\REMINST
Drag all the files from the REMINST folder in to the c:\HPCPackInstall folder, and for any 'Copy File' dialogs that come up select "Copy and Replace" and in any 'Confirm Folder Replace' dialogs select 'Yes' to 'Merge' the folders.
1. If there were an 'integration packs' available for download, extract the contents of the archives and merge them in to your new installation folder (i.e. c:\HPCPackInstall) in the order that they were release

Note:

The SP1 installation folder that you create using this method can only be used to install a head node of the same 'Edition' as what the head node in step 3 was running - So if your Step 3 headnode is running 'Express' this install folder can only produce 'Express' head nodes
This procedure only works with HPC Pack 2008 R2 Service Packs, it does not work with non-service pack updates (aka patches/hotfixes/qfes)

Microsoft iSCSI Software Target 3.3 for Windows Server 2008 R2 available for public download

Don Pattee — Wed, 25 May 2011 18:53:30 GMT

Hey HPC admins, since some of you are using iSCSI in your clusters here is some information that you may find interesting. It is copied from Jose Baretto's blog.

Introduction

For the last few years, I’ve been blogging about the Microsoft iSCSI Software Target and its many uses related to Windows Server Failover Clustering, Hyper-V and other server scenarios. Today, Microsoft has made this software publicly available to all users of Windows Server 2008 R2.

The Microsoft iSCSI Software Target has been available for production use as part of Windows Storage Server since early 2007. It has also been available for development and test use by MSDN and TechNet subscribers starting in May 2009. However, until now, there was no way to use the Microsoft iSCSI Software Target in production on a regular server running Windows Server 2008 R2. This new download offers exactly that.

Now available as a public download, the software is essentially the same software that ships with Windows Storage Server 2008 R2. Windows Storage Server 2008 R2 and the public download package will be refreshed (kept in sync) with any software fixes and updates. Those updates are described at http://technet.microsoft.com/en-us/library/gg232597.aspx.

This release was preceded by intense testing by the Microsoft iSCSI Target team, especially in scenarios where the iSCSI Target is used with Hyper-V and with Windows Server Failover Clusters. We do imagine these to be amongst the most commons deployment scenarios.

Testing included running the Microsoft iSCSI Software Target in a two-node Failover Cluster and configuring 92 individual Hyper-V VMs, each running a data intensive application and storing data on a single node of that iSCSI Target cluster. The exciting part of the test was to force an unplanned failure of the iSCSI Target node being used by all the VMs and verify that we had a successful failover to the other node with all 92 VMs continuing to run the application without any interruption.

How to download and install

To download the Microsoft iSCSI Software Target 3.3 for Windows Server 2008 R2, go to http://www.microsoft.com/downloads/en/details.aspx?FamilyID=45105d7f-8c6c-4666-a305-c8189062a0d0 and download a single file called “iSCSITargetDLC.EXE”. (Note: This was just released at 10AM PST on 04/04/2011, so the download might still be replicating to your closest download server. If the link does not work, try again later). This is a self-extracting archive that will show this screen when run:

Select a destination folder and click “Install”. Once it finishes, you will find a few files available to you in that folder:

Files	Description
<folder>\autorun.inf <folder>\bkgd-page-servers-dynagrid.jpg <folder>\copyright.jpg < folder>\Documentation <folder>\Index.htm <folder>\wsr2logo.bmp	Welcome page. Start with Index.htm
<folder>\Documentation\iSCSItarget_Gstart_R2.doc	Getting Started Guide
<folder>\Documentation\iSCSI_33_relnotes.doc	Release Notes
<folder>\x64\iscsitarget_public.msi	Microsoft iSCSI Software Target installer (64-bit only)
<folder>\x64\iscsitargetClient_public.msi	VSS and VDS providers for the iSCSI Target (64-bit)
<folder>\x86\iscsitargetClient_public.msi	VSS and VDS providers for the iSCSI Target (32-bit)

If you click on the index.htm file on the main folder, you will see the welcome page with a few links to the items included:

To install the iSCSI Target on a computer running Windows Server 2008 R2, simply run the “iscsitarget_public.msi” MSI file from a command line or right-click it on Windows Explorer and choose “Install”.

Frequently Asked Questions (FAQ)

Q: Can I install the Microsoft iSCSI Software Target 3.3 on Windows Server 2008 or Windows Server 2003?
A: No. The Microsoft iSCSI Software Target 3.3 can only be installed on Windows Server 2008 R2.

Q: Can I install the Microsoft iSCSI Software Target on Windows Server 2008 R2 with Service Pack 1 (SP1)?
A: Yes. In fact, that’s what is recommended.

Q: Can I install the Microsoft iSCSI Software Target on a Core install of Windows Server 2008 R2?
A: No. The Microsoft iSCSI Software Target 3.3 is only supported in a Full install.

Q: I don’t have a copy of Windows Server 2008 R2. Where can I get an evaluation copy?
A: You download an evaluation version of Windows Server 2008 R2 with Service Pack 1 from http://technet.microsoft.com/en-us/evalcenter/dd459137.aspx

Q: Where is the x86 (32-bit) version of the Microsoft iSCSI Software Target 3.3?
A: The Microsoft iSCSI Software Target 3.3, is provided in only in an x64 (64-bit) version, as is Windows Server 2008 R2,

Q: What are these “iSCSITargetClient” MSI files included in the download?
A: Those are the optional VSS and VDS providers for the Microsoft iSCSI Software Target 3.3. You should install them in the same computer that runs the iSCSI Initiator if you intend to use VSS or VDS. For details on VSS, see http://blogs.technet.com/b/josebda/archive/2007/10/10/the-basics-of-the-volume-shadow-copy-service-vss.aspx. For details on VDS, see http://blogs.technet.com/b/josebda/archive/2007/10/25/the-basics-of-the-virtual-disk-services-vds.aspx.

Q: Where is the Windows Storage Server 2008 R2 documentation?
A: There is some documentation inside the package. Additional documentation is available on the web at http://technet.microsoft.com/en-us/library/gg232606.aspx

Q: Can I use the Microsoft iSCSI Software Target 3.3 as shared storage for a Windows Server Failover Cluster?
A: Yes. That is one of its most common uses.

Q: Can I install the Microsoft iSCSI Software Target 3.3 in a Hyper-V virtual machine?
A: Yes. We do it all the time.

Q: Can I use the downloaded Microsoft iSCSI Software Target 3.3 in my production environment?
A: Yes. Make sure to perform the proper evaluation and testing before deploying any software in a production environment. But you knew that already…

Q: What are the support policies for the Microsoft iSCSI Software Target 3.3 on Windows Server 2008 R2?
A: The support policies are listed at http://technet.microsoft.com/en-us/library/gg983493.aspx

Links

I would recommend that you download and read my previous blog posts about the Microsoft iSCSI Software Target. Here are some of the most popular ones.

Please keep in mind that some of these posts mention previous versions of the Microsoft iSCSI Software Target that ran on different Windows Server versions. The overall guidance, however, still applies.

Conclusion

I hope you are as excited as we are about this release. Download it and experiment with it. And don’t forget to post a comment about your experience or send us your feedback.

'Dryad' becomes 'LINQ to HPC' !

Don Pattee — Fri, 20 May 2011 06:00:00 GMT

A few weeks ago we announced the beta release of Dryad, a capability for big data running on Windows HPC Server 2008 R2 and Microsoft HPC Pack 2008 R2-based clusters with Service Pack 2.

There's now a white paper by David Chappell that explains the newly-named LINQ to HPC in some detail:

Introducing LINQ to HPC: Processing Big Data on Windows

This paper provides an overview of the big data capability provided by LINQ to HPC. High performance computing jobs today are most often CPU-bound, and so the goal of running on a cluster is to let them use many CPUs at once. Big data jobs, by contrast, are typically I/O bound—they’re limited by how fast they can read and write data. Rather than relying primarily on processing power, these jobs most often do relatively simple computations on massive amounts of information. Running big data jobs on a cluster still makes sense, but the goal isn’t to use lots of CPUs at once—it’s to use lots of disks at once. When a single application can read from many disk spindles simultaneously, processing that data gets much faster. LINQ to HPC provides a platform for developers to build applications that can process large amounts of unstructured data. LINQ to HPC is part of Windows HPC Server 2008 R2 SP2.

Definitely worth a read if you're considering evaluating LINQ to HPC. Once you’ve read the paper and want to look at some examples of LINQ to HPC then the beta download includes dozens of examples of LINQ to HPC queries.

You can also watch Saptak Sen's talk from TechEd 2011, Running "Big Data" Applications on a Windows HPC Server Cluster on Channel 9.

Dryad Beta Program Update

Don Pattee — Fri, 06 May 2011 17:30:00 GMT

We’re happy to announce that as part of the beta for Microsoft HPC Pack 2008 R2 SP2 we’re shipping a beta of the project code-named “Dryad.” Dryad is Microsoft’s solution for “Big Data”. What’s Big Data? Today’s environment is full of ever growing mountains of data. From web logs and social networking feeds to fraud detection and recommendation engines or large science and engineering problems like genomic analysis and high energy physics. Big data is becoming a more and more common scenario. .

Dryad is designed to help you ask questions of big, unstructured, data. It allows you to distribute your data across an HPC cluster built from commodity hardware and execute queries against that data. Dryad is designed to achieve the best possible performance by exploiting data locality. Data locality means that Dryad executes query logic as close as possible to the data, rather than moving the data to the logic.

Dryad’s Distributed Storage Catalog (DSC) maintains a list of where data is distributed across the cluster and works in concert with the HPC Scheduler and Dryad’s Graph Manager to maximize data locality as the query executes.

Let’s take a quick look at the key features of Dryad…

Integration with Windows HPC Server

Dryad is part of HPC Pack 2008 R2 SP2 Enterprise. It installs as part of the HPC Pack on your cluster and you configure which nodes are storing Dryad data and you’re done. Dryad jobs are managed through the HPC Cluster Manager, just like MPI and SOA jobs. In addition there are command line tools to help administrators move data onto the cluster and to manage data.

For example suppose you had a very large collection of documents and wanted to look for the most common words within those documents. First you could use Dryad’s built in data administration looks to load data from your MyBooks folder on a share into Dryad:

DSC FILESET ADD \\MyServer\Shared\MyBooks MyWordData /service:MyHeadNode

As the data is loaded DSC distributes it across the cluster. You can use commands like DSC FILESET LIST to examine you data or view the DSC file set using the Windows Explorer.

A Familiar Programming Model with LINQ

Now the data is loaded how do I count the words? Dryad uses a LINQ based programming model to enable developers to express their queries using a familiar syntax. Behind the scenes Dryad does all the heavy lifting of creating a query plan, deploying your assemblies and executing the query in a scalable and robust way.

If you’re familiar with LINQ then writing DryadLINQ queries is very straightforward. Simply create an HpcLinqContext instance and execute a query.

Now from within a Visual Studio C# project create a HpcLinqContext that refers to your cluster head node:

  HpcLinqConfiguration config = new HpcLinqConfiguration("MyHeadNode");

  HpcLinqContext context = new HpcLinqContext(config);

Next create a query:

IQueryable<Pair> results = context.FromDsc<LineRecord>(inputFileSetName)
    .SelectMany(l => l.Line.Split(new []{ ' ', '\t' },
                                 StringSplitOptions.RemoveEmptyEntries))
    .GroupBy(word => word)
    .Select(word => new Pair(word.Key, word.Count()))
    .OrderByDescending(pair => pair.Count)
    .Take(200);

Dryad leverages the LINQ programming model rather than forcing you to rethink your query in terms of a particular pattern like MapReduce.

Finally print the results. This will cause the query to be executed on the cluster.

  foreach (Pair result in results)

    Console.WriteLine("  {0,-20} : {1}", result.Word, result.Count);

As you’re job runs you can use the HPC Job Manager to view its progress, just like any other job running in HPC Server.

That’s it! Distibuted word count using Dryad in just a few lines of code. The word count example is just one of several samples we’re shipping with the beta to get you started. Others include; sort, k-means clustering, table joins and SQL connectivity to mention just a few.

Dryad Leverages Existing Proven Microsoft Technologies

While Dryad provides new features to address the challenges of Big Data it’s important to understand that it’s built on a series of proven technologies and was developed by Microsoft Research over several years. It has already been used internally, at scale within Microsoft on several large projects. It’s build on a series of mature technologies like NTFS and SQL Server and integrates with other parts of your enterprise infrastructure like Active Directory and builds on existing .NET tools like Visual Studio.

Please download and evaluate the HPC Pack 2008 R2 SP2 Beta and the accompanying documentation and samples. We’d be very pleased to hear your feedback on our DevLabs Forum.

HPC Pack 2008 R2 SP2 Beta Program

Don Pattee — Mon, 02 May 2011 21:46:00 GMT

HPC Pack 2008 R2 SP 2 Beta is a pre-release version of the upcoming service pack which will include a number of new features such as:

Ability to use Virtual Machines in Azure
Ability to run MPI-based applications in Azure
Linq to HPC (formerly known as 'Dryad')
Support for 'workstation nodes' to be in a different domain from the dedicated cluster hardware
A preview of our new web-based portal
For more details please review the 'What's New' guide

Head over to http://connect.microsoft.com/HPC to try it out.

Are you running SP1 yet?

Don Pattee — Fri, 25 Mar 2011 22:30:56 GMT

If you have an HPC Pack 2008 R2-based cluster that you have not yet upgraded to SP1 - Why?

HPC Pack service packs provide the most current set of fixes to improve stability, reliability, and performance. This particular service pack also included some new features, such as the ability to run certain workloads in Azure (http://technet.microsoft.com/en-us/library/gg481751(WS.10).aspx)

Let us know your thoughts over on our discussion forum: http://social.microsoft.com/Forums/en/windowshpcitpros/thread/205ef11d-d71a-46cc-830d-234b1436c1df

power management solution in Windows HPC Server 2008 R2 (SP1) Monitoring Management Pack

Yang Zhang - MSFT — Tue, 15 Mar 2011 07:48:05 GMT

Summary

As Windows HPC Server 2008 R2 (SP1) Green IT offering, we have enabled the power management solution in Windows HPC Server 2008 R2 (SP1) Monitoring Management Pack with two configurable rules: “Calendar-based Power Management Rule” and “Consumption-based Power Management Rule”.

· Calendar-based Power Management Rule

With calendar-based power management rule, you are able to define a certain time period in a day when you want a specified portion of the compute nodes going hibernated to save the power; and also you can define this policy only applies for certain days in a week.

· Consumption-based Power Management Rule

With consumption-based power management rule, we are able to evaluate the cluster utilization over a time period with the number of queued jobs and make the decision on whether we should hibernate a portion of the compute nodes to save the power.

We have defined three levels of cluster capability and each time when the hibernate condition is met, the cluster will change from current capability level to a lower level; and on the other hand, when the wake up condition is met, the cluster will change from current capacity level to a higher level.

Configure the Rules

Both of the above rules are disabled by default, admin is able to enable them and configure them easily after importing Windows HPC Server 2008 R2 Monitoring Management Pack into SCOM server.

Open “System Center Operations Manager”, go to “Authoring” wunderbar, select “Rules”, look for keywords “Power Management”, then you can find these two rules listed as following.

Here lists a set of important configurations you are able to override for the two rules and their default value:

Calendar-based Rule
Parameter Name	Default Value	Notes
Enabled	False	The rule is disabled by default
Start Time	0:00	The time each day when power-saving mode for compute node starts.
End Time	6:00	The time each day when power-saving mode for compute node ends.
Exclude Days		A list of days each week when compute nodes are excluded from entering power-saving mode. The “exclude days” format is like: “Saturday, Sunday”
Power On Percentage	70	The percentage of compute nodes that will remain power on during the power-saving mode
Consumption-based Rule
Parameter Name	Default Value	Notes
Enabled	False	The rule is disabled by default
HighCapacityLevel	100	The percentage of high compute node capacity definition
MediumCapacityLevel	80	The percentage of medium compute node capacity definition
LowCapacityLevel	60	The percentage of low compute node capacity definition
UpperQueueLength	5	The length of the job queue above which the rule can cause the compute nodes to reach a higher capacity level
LowerQueueLength	1	The length of the job queue below which the rule can cause the compute nodes to reach a lower capacity level
LowConsumption	40	The compute node consumption percentage below which the rule can cause the compute nodes to reach a lower capacity level
Number of Samples	6	The number of samples to identify the LowConsumption which can push the compute nodes to enter a lower capacity level, the sampling interval is following “interval seconds”
Interval Seconds	300	The sampling interval, default is 300 seconds.

Power Saving Evaluation

To evaluate the Power saving efficiency and the impact on the job throughput, we conducted the following experiment:

(1) Setup an HPC cluster with 1 Head node, 1 broker node and 4 compute nodes.

(2) Setup the job submission simulation in one typical working day as following:

Also the job length is distributed as following:

(3) Compare the power saving efficiency and also the impact to job throughput for following three sceanrios:

a. Disable the power management rules;

b. Enable only the calendar-based power management rule;

c. Enable only the consumption-based power management rule.

Note:

We adjusted a little bit the configurations for both rules in the experiment:

· Calendar-based rule:

o Set StartTime to 22:00, EndTime to 7:00, PowerOnPercentage to 60%.

· Consumption-based rule:

o Set UpperQueueLength to 2.

Here are the experimental results:

· Power saving efficiency.

We use the # of hibernated nodes multiply the period of time to evaluate the power saving efficiency. By applying calendar-based rule, there are 2 nodes hibernated from 22:00 to 7:00, while applying consumption-based rule, 2 nodes get hibernated from 21:00 to 9:00. Both rules have saved some power for the cluster, while consumption-based rule worked better than calendar-based rule.

· Utilization on available cores

Consumption-based power management rule has achieved the highest utilization on available cores (49.1%), followed by calendar-based rule (47.1%) and no rule enabled (42.7%).

· Impact to job throughput

Job throughput measures the average number of completed jobs per hour for a day, and the throughput is the same for all three scenarios.

· Impact to job turnaround

Job turnaround measures how much time a job needs to wait compared to how much time it runs. Job turnaround increases a little bit for consumption-based rule (from 0.436 to 0.437), but very minimum; it remains the same for calendar-based rule and no rule enabled scenario.

Conclusion

Based on above evaluation, the power management rules are able to save the power effectively for the cluster without bringing noticeable impact on job throughput and job turnaround. It is able to help you achieve the Green IT goal for your cluster. J

A few HPC updates for the week

Don Pattee — Fri, 04 Mar 2011 19:49:00 GMT

For everyone who loves the Excel functionality in the HPC Pack 2008 R2 Enterprise version, we have written up some best practices for using the HPC macros.

Then we have released an update to the Network Troubleshooting Report portion of the tool pack. This is a minor bug fix for people who are using Server 2008 on some of their compute nodes. You can download the updated version on the HPC Pack 2008 / 2008 R2 Tool Pack page.

And, we also released an update (aka qfe/hotfix/patch) for folks using HPC Pack 2008 R2 Service Pack 1. The update fixes two problems

1) Submitting a job from a user account in a domain that does not have a trust relationship with the headnode’s domain will fail. After applying this fix (to the cluster and the client) the submitter can enter appropriate credentials via the Job Console or ‘runas’ from command line / powershell.

2) For clusters that allow jobs to be preempted there is a case where a heavily stressed head node can receive a request from a user to cancel a job at the same time it was trying to preempt that job can then it reports to the user that the cancellation occurred, but the actually got preempted/requeued.

You can download this update on this page: http://www.microsoft.com/downloads/en/details.aspx?FamilyID=d4982499-45e6-4db5-a490-ff14bec277f8

If you have questions/comments on any of those items please hop over to the HPC discussion forums.

HPC documentation updates

HPC doc team — Tue, 22 Feb 2011 01:45:05 GMT

We've published some new topics to the tech net library and the download center recently. Here are the links:

· Replacing a Head Node Configured in a Failover Cluster in Windows HPC Server 2008 R2

· V3 glossary

· New developer landing page on MSDN

· Debugger docs for VS2010 add-in

· AppFabric caching whitepaper on the download center

· SP1 Azure Deployment doc now available as a download

If you have questions, suggestions, requests, or feedback about these topics or Windows HPC documentation in general, please comment on this post or email the HPC doc team at hpcv3doc@microsoft.com.

Thanks!

New HPC Sample Code

Don Pattee — Fri, 04 Feb 2011 19:06:26 GMT

Today we have added an example of how to use Windows Server AppFabric Caching in conjunction with high performance SOA-based applications. The 'Common Data Caching Sample' package contains both a white paper describing the solution as well as some sample code showing how to inject and consume data from the in-memory cache.

Last month I believe I neglected to blog about the 'Silverlight Reporting Sample' that was added. It shows how to use our 'reporting extensibility' feature to create custom reports, which can then be made available to admins and users through a fancy Silverlight web app.

Both are available on the HPC Pack 2008 R2 SP1 SDK page. For questions/discussion on these samples, or to suggest a topic for our next sample code update, head over to the Windows HPC Developer Forum.

HPC Java Bridge - Enabling Java-to-Java Communication

Torsten Langner — Mon, 31 Jan 2011 08:54:00 GMT

HPC Server 2008 R2 SDK SP1 contains sample code that demonstrates the usage of Java on the client side. The framework enables a Java-to-Java communication within HPC Server 2008 R2 SP1. It enables to send Java objects on the client side and process them on the service side within stateful Java processes. The Project Sources are published on MSDN.

Architectural Overview - How Does it Work?

The HPC Java Bridge plugs into the existíng HPC SOA architecture. We can identify at least 3 different tiers that are related to the bridge:

The Java Client: On the client side a programmer can serialize Java objects that are sent via the bridge to the Java service side. The client uses the Java API parts of the proposed framework to create a session pool based on durable or non-durable sessions. The framework further allows to transfer the serialized messages in a pretty easy way to the Java service side. The client API parts contain failover handling and a support of large message transfers. Processed messages or exceptions are forwarded asynchronously.
The HPCJavaBridge.dll: The HPCJavaBridge is a standard WCF service that acts as a gateway for Java objects. The service spawns Java processes that implement the service parts of the Java API and controls their lifecycles. It keeps the spawned Java processes up and running for the lifetime of the WCF service instance.
The Java Service: On the service side a programmer needs to extend the service parts of the proposed Java API. Lifetime and exception handling of the code are handled by the framework.

Configure the WCF Service

Please follow these steps to configure the WCF Service.

Create a new WCF service configuration file called HPCJavaBridge.config or edit the one provided in the Project Sources. Edit the ServiceRegistration section of the configuration file and insert the path to the HPCJavaBridge.dll assembly.
Specify the ChunkFolder location: Serialized Java objects > 3 MB (durable mode) or > 35 MB will be chopped into chunks. The chunks must be stored on a UNC path location.
Specify the location of java.exe on the compute nodes using the JavaExe environment variable. The WCF Service will spawn a Java process and forward the incoming serizalized Java objects.
Specify the Java arguments using the JavaArgs environment variable. This is the location where to configure the Java classpath and the Java Service entry point. In the provided sample it's the class hpc.test.PingPongSampleService. Do not add the Apache CXF libraries to the specified classpath.
Specify the PortRangeStart and the PortRangeEnd environment variables. The spawned Java process will listen to a random port within the port range on localhost of the executing compute node. Make sure that the firewall settings on the compute nodes won't disturb this communication gateway.
Run the SOA Service Loading Test against the new WCF service from the Cluster Manager console.

<microsoft.Hpc.Session.ServiceRegistration> <service assembly="\\myshare\HPCJavaBridge\HPCJavaBridge\bin\Debug\HPCJavaBridge.dll" contract="HPCJavaBridge.IHPCJavaBridge" type="HPCJavaBridge.HPCJavaBridge" includeExceptionDetailInFaults="true" maxConcurrentCalls="0" serviceInitializationTimeout="60000" stdError="" maxMessageSize="2147483647"> <environmentVariables> <add name="ChunkFolder" value="\\myshare\Chunks\"/> <add name="JavaExe" value="\\myshare\jdk1.6.0_21\jre\bin\java"/> <add name="JavaArgs" value="-Xms64m -Xmx1024m -cp \\myshare\workspace\HPCJavaService\bin\. hpc.test.PingPongSampleService"/> <add name="PortRangeStart" value="42000"/> <add name="PortRangeEnd" value="49000"/> </environmentVariables> </service> </microsoft.Hpc.Session.ServiceRegistration>

Implementing a Simple Client

The Java client side requires the CXF libraries as well as the generated Session and HPC Java Bridge service API. Once added to the classpath create custom Java client code as displayed below. It allows compression, serialization and distribution of any serializable Java object through the JavaClientSingleton class.

Let's have a look at the following SimplePingPongClient sample:

Creating a session in order to connect to a SOA service in HPC Server 2008 R2 requires at least the specification of the name of the head node, the SOA service name and the AD user details. The JavaClientSingleton class further allows to specify a job template name (if not it will use the default job template) and the minimum and maximum required cores. The session share feature allows multiple broker clients to submit their messages through the same session. This speeds up the communication between the client and the service because the session creation time is quite long compared to the submission time of a message. If a durable behavior is required, the JavaClientSingleton will create durable sessions instead of non-durable ones.
If messages or exceptions from the service side are received, the modified session API of HPC Server 2008 R2 will invoke the OnMessageReceive-method of an AsyncMessageReceiver object. The object received is already de-compressed and de-serialized.
To send messages from the Java client to the Java service side the JavaClientSingleton class provides the distributeJobs-method that compresses and sends an array of serializable object to a(n existing) session.

package hpc.test;

import com.microsoft.hpc.scheduler.session.modified.AsyncMessageReceiver;
import hpc.clientSide.JavaClientSingleton;
import hpc.sharedObjects.PingPong;

public class SimplePingPongClient {

public static void main(String[] args) throws Exception {
   JavaClientSingleton jc = JavaClientSingleton.getInstance();
   jc.setHeadNode("head");
   jc.setDomainName("hpc");
   jc.setUserName("alibaba");
   jc.setPassword("al!b!ba");
   jc.setServiceName("HPCJavaBridge");
   jc.setJobTemplateName("hpcjavabridge");
   jc.setMinUnits(2);
   jc.setUnitType(0);
   jc.setMaxUnits(4);
   jc.setReuseSession(true);
   jc.setUseDurableSessions(false);

   AsyncMessageReceiver amr = new AsyncMessageReceiver() {
      @Override public void OnMessageReceive(Object Message) {
          System.out.println("[received]");
          if(Message instanceof PingPong)
          {
             PingPong pp = (PingPong) Message;
             System.out.println("ping/pong from host: " + pp.getHostName());
          }
    }
   };

   jc.distributeJobs(new Object[] {new PingPong(), "I'm a string job", new Long(123)}, 4, amr, true);
  }
}

Method	Description
setHeadNode	sets the name of the cluster head node
setDomainName	sets the name of the users domain
setUserName	sets the username
setPassword	sets the users password
setServiceName	sets the service name under which the HPCJavaBridge got configured
setJobTemplateName	sets the name of the job template. If non is specified it will use the Detault job template
setMinUnits	sets the guaranteed minum amount of units. The value must be > 0
setMaxUnits	sets the maximum amount of units. The value must be bigger or equal than the minimum amout of units
setUnitType	sets the unit type to allocate on the cluster. Allowed values are: 0 = core, 1 = socket, 2 = node
setReuseSession	if set to true sessions won't be closed if possible. This reduces the the send/response time for new job distributions
setUseDurableSessions	if set to true only durable sessions will be used for job distribution. In case of a failover the distributed jobs won't be lost
distributeJobs	distributes the job objects. Parameters are: Object[] jobs: the array of serializable job objects that will be sent to the cluster int priority: the session priority. Allowed values range from 0 (= lowest priority) to 4 (= highest priority) AsyncMessageReceiver clientCallback: the async callback object boolean waitForCompletion: if set to true the method won't return before all distributed jobs responses are collected
warmUp	Warms up a session by sending simple ping pong messages to the cluster. This will create a session and initialize the service instances which consumes most of the time. By calling this method at the beginning of all user interactions the minimum send/response time is achieved. Parameters are: int priority: the session priority. Allowed values range from 0 (= lowest priority) to 4 (= highest priority) AsyncMessageReceiver clientCallback: the async callback object boolean waitForCompletion: if set to true the method won't return before all distributed jobs responses are collected
closeAllSessions	Closes all sessions.

Implementing the Service Side

The published service API includes a ping pong test class. Any Java service that will be implemented to interchange with the proposed HPC Java Bridge framework will need to extend the JavaProcessStarterImpl class. Place your custom code with these 3 methods:

The initializeInstance-method is called first by the HPC Java Bridge framework when the Java process got spawned by the WCF service. Place all initialization code in here since it will be called only once per JVM instance.
The shutdownInstance-method is called first by the HPC Java Bridge framework when the associated SOA session is finishing. Place all instance cleaning code in here since it will be called only once per JVM instance.
The processInstance-method is called every time a Java object is arriving from the client side. Place your custom message processing code in here.

Make sure to add a main-method to your starting class. Call the Publish-method to startup the Java process to start the communication with the HPC Java Bridge framework. The WebService attribute values need to be identical for every custom service. Make sure to copy and past them from the PingPongSampleService class.

package hpc.test;

import hpc.serviceSide.fileLessIntegration.JavaProcessStarterImpl;
import java.net.InetAddress;
import java.util.Date;
import javax.jws.WebService;

@WebService(
endpointInterface = "hpc.serviceSide.fileLessIntegration.JavaProcessStarter",
name="JavaProcessStarter",
serviceName="JPSService",
targetNamespace=http://hpcjavasoabridge.org/soa2java)

public class PingPongSampleService extends JavaProcessStarterImpl {

  @Override protected void initializeInstance() {
      System.out.println("local init");
  }

  @Override protected void shutdownInstance() throws Exception {
      System.out.println("good-bye!");
}

   @Override protected Object processInstance(Object JavaObject) throws Exception {
      System.out.println("processing object");
      if(JavaObject instanceof Long)
         return "hello long";
      else
         return new Date();
  }

public static void main(String[] args) throws Exception {
       PingPongSampleService PPSS = new PingPongSampleService();
       PPSS.Publish(args);
}

}

Performance

The performance tests are ongoing. First results with sending, compressing, receiving and decompressing simple PingPong Java objects within a "warm" session show very, very fast results.

Run Batch Workload on a Mixed Infrastructure (Windows Azure Worker Nodes & On-Premise HPC Server 2008 R2 Compute Nodes)

Torsten Langner — Sun, 16 Jan 2011 04:05:47 GMT

With the introduction of SP1 of HPC Server 2008 R2 it is possible to run workload on Windows Azure. If you want to be able to off-burst your batch application (= extending the infrastructure from classic on-premise servers to cloud based Azure worker nodes) you need to set specific environment variables within the on-premise server infrastructure.

Here is a sample of how to upload a batch based application (azuretestbatch.exe) to Azure workder nodes and run a parametric sweep workload across the full cluster (on-premise compute nodes + Azure worker nodes):

Create the Windows Azure Package & Upload It

Use the hpcpack create command to create a *.zip file. Once the package got created, upload it to the Windows Azure storage by using the hpcpack upload command. Specify the node template of your Windows Azure worker nodes. Additionally specify a relative path – in our sample its named test. This forces the hpc sync command which we call later to unzip the package content to the folder named test:

C:\Users\tlangner>hpcpack create AzureTestPackage.zip
C:\temp\folder1

C:\Users\tlangner>hpcpack upload AzureTestPackage.zip
/nodetemplate:AzureNodeTemplate /relativepath:test

You can test your upload by calling the hpcpack view command. As you can see in the lines below the targeted physical directory on the Windows Azure worker node disks is %CCP_PACKAGE_ROOT%\test. The environment variable %CCP_PACKAGE_ROOT% only exists on Windows Azure worker nodes.

C:\Users\tlangner>hpcpack view azuretestpackage.zip
/nodetemplate:AzureNodeTemplate
Connecting to head node: head
Querying for Windows Azure Worker Role node template "AzureNodeTemplate"
Windows Azure Worker Role node template found.
Retrieving Azure account name and key.
Found account: hpc*** and key: ********
Package Name:           azuretestpackage.zip
Uploaded:               06.01.2011 10:43:44
Description:            Node
Template:               AzureNodeTemplate
Target Azure Dir:       %CCP_PACKAGE_ROOT%\test

After the hpcpack upload command has finished, it is time to deploy the package content to the Windows Azure worker nodes (ie. local disks of the cloud machines). This is done by calling the hpcsync command:

C:\Users\tlangner>hpcsync /nodetemplate:AzureNodeTemplate

Deploy the Package Content Locally

In order to run the batch workload on the “classic” compute nodes, too, it is helpful to deploy the package using the same environment variables as they are are available in Windows Azure. In this sample we want the CCP_PACKAGE_ROOT variable link to the local C:\TEMP folder of every machine in the cluster:

C:\Users\tlangner>cluscfg setenvs "CCP_PACKAGE_ROOT=C:\TEMP"

Then, let’s copy the zip package content to the target directory of every “classic” compute node in the cluster. The content will be deployed to the C:\TEMP\test directory:

C:\Users\tlangner>clusrun /nodegroup:computenodes xcopy
\\head\temp\AzureTestBatch\AzureTestBatch\bin\Debug\*.*
^%CCP_PACKAGE_ROOT%^\test

Run the Mixed Workload

After deploying the package content to the whole cluster it’s time to run the batch job:

C:\Users\tlangner>job submit /nodegroup:X
/parametric:1-1000 /workdir:^%CCP_PACKAGE_ROOT%^\test
azuretestbatch.exe 5000