HPC, Virtualization and Random Thoughts

Using CloudBlitz to Submit Jobs

gmarchetti — Thu, 16 May 2013 21:22:00 GMT

In the second post about Cloudblitz (http://cloudblitz.codeplex.com) we'll examine how to use it to submit jobs to a deployed Azure cluster.

Keep in mind that in Microsoft's HPC scheduler implementation, jobs are just containers. They contain one or more tasks to be executed and have properties associated to them (e.g. credentials, priority).

Such a container can be created empty, then filled with tasks and finally submitted for execution. If you have a look at the Create-InstallJob.ps1 script that comes with Cloudblitz, you'll see an example of that.

Let us suppose that you want to write a simple job submission script:

1. Import the Cloudblitz.Powershell module and get the hpc object types:

Import-MyModule -Name "CloudBlitz.Powershell" -Path $cmdletsPath
$hpcTypesPath = [System.IO.Path]::GetDirectoryName($cmdletsPath) + "\HpcSchedulerManagement.dll"
Add-Type -Path $hpcTypesPath -ErrorAction Stop

2. Provide the credentials for the jobs to be submitted

# Fill in hpc scheduler credentials
$hpcCredentials = New-Object CloudBlitz.Powershell.Cmdlets.HpcSchedulerCredentials
$hpcCredentials.ClusterName = $ClusterName
$hpcCredentials.AdminLogin = $AdminLogin
$hpcCredentials.AdminPassword = $AdminPassword
$hpcCredentials.CertificateThumbprint = $mgmtcert.Thumbprint

Note that you will need the thumbprint of the certificate associated with the service at deployment. That is required because the powershell commandlets use the hpc scheduler REST API to communicate with the service. All the parameters can be passed on the command line or retrieved from the ConfigurationVariables.ps1 file that you edited before deployment.

3. Create an empty job container

# Create the job on the scheduler
$Job = New-Object HpcSchedulerManagement.DataContracts.SchedulerJob
$Job.FailOnTaskFailure = $True
$Job.Name = "Job"+$(get-random 1000000)
$Job.Id = Add-SchedulerJob -Credentials $hpcCredentials -Job $Job -ErrorAction Stop

The first line creates an empty container, then we set some properties (e.g. random name) and finally we create the job on the hpc scheduler and retrieve its id.

4. Add tasks into the job container

# Create the task
$Task = New-Object HpcSchedulerManagement.DataContracts.SchedulerTask
$Task.Name = "Task"+$(Get-Random 1000000)
$Task.CommandLine = $CommandLine
$Task.Id = Add-SchedulerTask -Credentials $hpcCredentials -JobId $Job.Id -Task $Task

As with the job, we create an empty object, set its properties, then add it to the job container whose id we retrieved before. Note that one of those properties is $Task.CommandLine, which takes a string. The string is an arbitrary command to be executed on the remote nodes. If you want to start a MPI job, it will look like

mpiexec <mpi parameters> <your mpi>.exe <app parameters>

5. Submit the job for execution

# Submit the job
Write-Host "Submitting the job"
Submit-SchedulerJob -Credentials $hpcCredentials -JobId $Job.Id

# Wait for the job to complete
Write-Host "Waiting for job $($Job.Id) to complete"
$finalState = Wait-SchedulerJob -Credentials $hpcCredentials -JobId $Job.Id

Write-Host "Job $finalState"

Note that you need not wait for the job to complete. Those lines just show a way to retrieve and display the final job state. Normally, you will submit, then query for status later with something like:

Get-schedulerjob -Credentials $hpcCredentials -JobId $JobId

That line will retrieve the job object and also display all its properties (not just those set in these samples).

You can retrieve the whole list of jobs in the scheduler with:

Get-schedulerjobList -Credentials $hpcCredentials

The job properties are those supported by Microsoft's HPC scheduler (http://msdn.microsoft.com/en-us/library/microsoft.hpc.scheduler.ischedulerjob_properties(v=vs.85).aspx).

Let's see an example of those, retrieved using Get-SchedulerJob:d : 8

Name : Job641676

UserName : topguy

Project :

RuntimeSeconds : 0

MinCores : 1

MaxCores : 1

MinSockets : 1

MaxSockets : 1

MinNodes : 1

MaxNodes : 1

UnitType : Core

RequestedNodes :

IsExclusive : False

RunUntilCanceled : False

NodeGroups :

FailOnTaskFailure : True

AutoCalculateMax : True

AutoCalculateMin : True

Preemptable : True

MinMemory : 0

MaxMemory : 0

MinCoresPerNode : 0

MaxCoresPerNode : 0

SoftwareLicense :

OrderBy :

ClientSource : AzureRestServiceAgent

Progress : 100

ProgressMessage :

TargetResourceCount : 0

ExpandedPriority : 2000

ServiceName :

NotifyOnStart : False

NotifyOnCompletion : False

EmailAddress :

Priority : Normal

Password :

JobTemplate : Default

JobType :

RequeueCount : 0

AutoRequeueCount :

RequestCancel :

Owner : topguy

SubmitTime : 5/15/2013 11:20:47 PM

CreateTime : 5/15/2013 11:20:46 PM

StartTime : 5/15/2013 11:20:48 PM

EndTime : 5/15/2013 11:20:50 PM

State : Finished

HoldUntil :

HasRuntime : False

CanGrow : True

CanShrink : True

PreviousState : Running

ChangeTime : 5/15/2013 11:20:50 PM

ExcludedNodes :

AllocatedNodes :

You will typically want to set at job submission MinCores and MaxCores (min. and max n. of cores required, default is 1). Note that properties are inherited by the tasks, unless overridden by those set at the task level (http://msdn.microsoft.com/en-us/library/microsoft.hpc.scheduler.ischedulertask_properties(v=vs.85).aspx). For MPI jobs the number of cores is fixed at submission, so MinCores=MaxCores.

Cloudblitz: A Tool to Deploy HPC Clusters in Azure

gmarchetti — Mon, 13 May 2013 18:31:00 GMT

I'm happy to announce the availability of Cloudblitz, a tool (framework) to deploy HPC clusters in Azure programmatically via Powershell, then submit jobs to them via Powershell as well.

You can find it at http://cloudblitz.codeplex.com

The tool builds on the HPC scheduler for Azure and the Powershell commandlets for Azure to solve a common problem, i.e. provide a scriptable way to create and interact with an Azure PaaS cluster. I refer you to the Codeplex page for the complete description and documentation. In this blog post, I'll cover how to build it and use it.

1. Make sure that you download and install all the dependencies listed on the codeplex page.

2. Download and extract the source code in its own directory.

3. Open a Visual Studio 2012 command prompt as administrator (dear old cmd.exe)

4. Go to the "BuildScripts" directory and run BuildCloudBlitz.cmd. This will build the required modules and copy the results in a "Packages" directory.

5. Open a Windows Azure Powershell prompt as administrator. Go to the CloudBlitz.ClusterTemplate directory

6. Edit the ServiceDefinition.csdef file (it is just an XML file). In particular note:

<WorkerRole name="ComputeNode" vmsize="Small">

...

<WebRole name="HeadNode" vmsize="Small">

Choose your VM size and type it in. In general, a small head node is sufficient (the head node does NOT run SQL, rather it uses SQL Azure, which reduces its memory requirements). If you're planning to run thousands of tasks at a time, go for a large instance.

7. Go to the CloudBlitz.Powershell\Scripts directory

8. Edit the ConfigurationVariables.ps1 file. In particular note:

$subscriptionName = "<your subscription name>"
$subscriptionId = "<your subscription id>"
$administratorName = "<admin name for the hpc cluster>"
$systemPassword = "<admin password for hpc cluster>"
$certificateFileNameRoot="<name of certificate>"
$deploymentName="<name of service>"
$deployLocation="<data center location>"
$HeadNodeInstanceCount = 1
$ComputeNodeInstanceCount = <n. of desired compute nodes>
$removePreviousDeploymentAndService = $true

9. Run InstallAndExportCertificate.ps1 This will create a .cer and a .pfx self-signed certificate file, install it in the local user certificate store.

10. Via the Azure portal, upload the .cer file as a management certificate.

11. Run the DeployDeployerRole.ps1 script. It will upload the packages, create an azure worker role that deploys a hpc cluster as per configuration, then remove the deployer role. Such a temporary role is necessary to avoid any firewall and connectivity issues. Most people will have ports 80 and 443 open for http and https. The Azure HPC scheduler deployment also requires a connection to SQL Azure on port 1433, which is normally closed. By letting the temporary role manage the process, such connections are all internal to Azure.

You can now connect to your cluster.cloudapp.net via RDP and start submitting jobs in the "traditional" way via the job manager or the command line.

Alternatively, you can use the set of powershell commandlets we developed from your workstation. They will be the subject of the next blog post :-)

Endpoints, firewalls and other annoyances

gmarchetti — Thu, 05 Jan 2012 22:22:49 GMT

When you deploy a HPC cluster on Azure, you typically want to run some application in it besides those provided in the azure samples. Those applications may require their own ports to be opened on the internal network and endpoints to be established for both internal and external communication. There is no hpc cluster manager gui in a pure azure implementation, so you'll have to perform such configuration manually.

1. Enabling endpoints

Endpoints must be configured prior to deployment, in the Azure service definition file. For instance, if you want to allow file sharing between nodes, you'll need SMB endpoints.

Note that the default configuration for azure hpc samples enables port 445 but not 139. That can be corrected via Visual Studio or by manually editing the .csdef file.

If you define a public endpoint too (as in the picture), remember that it will be subject to the azure load balancers and no "sticky" sessions are supported.

After deployment, you'll be able to share files and directories as usual. If you use a directory on an azure drive mounted as per my previous post, you will have a form of shared permanent storage for computation inputs and results that will persist between deployments.

2. Opening ports on the nodes

Establishing endpoints may not be sufficient. You will also need to open the relevant ports on the o/s firewalls of the nodes in question. For instance, the command line below will open ports 55000-55500:

clusrun /all netsh advfirewall firewall add rule name="myapp" dir=in protocol=tcp localport=55000-55500 action=allow

You may also need to register an exception with the windows firewall for the executables that you plan to run and any other executables that they may invoke in turn:

clusrun /all hpcfwutil register myapp.exe "d:\Program Files\myapp\myapp.exe"

At this point you can submit jobs via the portal or the command line, e.g.

job submit /numcores:16 /stdout:\\headnode1\share\myapp.out /stderr:\\headnode1\share\myapp.err mpiexec myapp.exe -<parameters>

The command line above submits an mpi job requiring 16 cores to run myapp.exe (assuming it is installed on the nodes), redirects outputs and errors to files on a share on the headnode.

3. Observations

Note that in the azure hpc sample deployment, the headnode is always called headnode1 and the compute nodes computenode1, 2, 3, etc... This has no relationship with the actual host names, which are assigned randomly by the azure fabric controller at deployment and can change. Also, computenode1 is not necessarily the first instance of the computenode role listed on the azure portal.

A list of aliases is maintained automatically in %WINDIR%\system32\drivers\etc\hosts

I have noticed however that for all mpi jobs I have submitted, mpi rank 0 was always on computenode1. That may be a sheer coincidence, as there is no reason for it to be there that I can think of. I'd be curious to know whether you find the same situation or not.

This may be useful anyway, because certain applications may require a connection to rank 0 for their GUI, in order to display the status of the running computation.

Powered by Qumana

Getting data and applications to / from Azure

gmarchetti — Tue, 13 Dec 2011 18:08:11 GMT

After Wenming’s post on deploying an azure hpc cluster with powershell, I have been looking for an easy way to transfer applications & data to / from such cluster.

I have found that in HPC SP3 the hpcpack command includes the option to upload and mount a VHD drive on azure nodes. The command is also available with the free client utilities. That is very useful because:

- You can put whatever you like in the vhd file, including the executable that you need to run.

- It is stored in blob storage, so it is permanent. If you need to delete the cluster deployment in order to save money, the data in the vhd drive will stay.

- The vhd drive can be snapshotted and the snapshots mounted read-only on multiple nodes.

- You can download the vhd back to your PC when you’re done and mount it.

Here’s a summary of the process:

1. On your PC, create and attach a fixed-size vhd drive with the storage management tools (GUI or diskpart).

2. Copy your files to the drive.

3. Again with storage manager gui or diskpart, detach the vhd.

4. Use hpcpack (or a gui tool like cloudberry explorer) to upload the vhd file to blob storage

- hpcpack upload <file>.vhd /account:<storage account name> /key:<storage key> /container:<container name> /description:<description string>

5. Connect via rdp to the headnode in azure (or any other node where you want to mount the drive).

6. To mount read-write on one node, run

- hpcpack mount <file>.vhd /account:<storage account name> /key:<storage key> /container:<container name>

7. To mount read-only, add the /snapshot parameter. You can also run the command on specific nodes via job manager or clusrun.

Once the drive is mounted, you can share it out via the GUI or with net share. The smb endpoints and firewall exceptions must be configured on all nodes for that to work.

When you’re done, you can use the same gui tool or hpcpack to download the vhd file back to your pc. The download does not erase the file from blob storage.

- hpcpack download <file>.vhd /account:<storage account name> /key:<storage key> /container:<container name>

Mount the vhd drive on your pc and retrieve your results.

SQL Server in Windows Azure

gmarchetti — Sat, 10 Sep 2011 06:08:36 GMT

It is certainly possible to run SQL Server 2008 R2 in Azure virtual machines, but keep in mind that they are not persistent between deployments, hence you want to use them for testing only and be aware of potential data loss.

In order to mitigate that risk you may want to set sql up in such a way that logs and data files are stored on Azure drives mounted by the VM – but that’s a topic for another blog post

Azure VMs must be uploaded in a sysprepped state (see previous post) for their deployment to succeed. Fortunately in SQL 2008R2 there is a way to “package” the application so that it can be installed after the virtual machine deployment. The trick is to automate the completion of such installation.

Here’s an outline of the process to follow:

1. Build a Sever 2008 R2 virtual machine in hyper-v

2. On that virtual machine, prepare a SQL Server 2008 R2 image for deployment with sysprep

a. copy the installation media on a local directory (e.g. c:\sqlinstall)

b. prepare the image as described here: http://msdn.microsoft.com/en-us/library/ee210664.aspx

c. Capture the sql server setup configuration file configurationfile.ini

3. Edit the configuration file to instruct sql setup to complete the installation as required. Keywords are:

a. ACTION="CompleteImage"

b. QUIET="True"

c. IACCEPTSQLSERVERLICENSETERMS="TRUE"

d. SECURITYMODE="SQL"

e. SAPWD="<insert your sa password>"

Remember that no active directory exists on Azure, so you will need to use SQL authentication and run the sql services under local credentials, e.g. “NT AUTHORITY\NETWORK SERVICE”. If you plan to connect to sql server, make sure that tcpip is enabled and the relevant port (1433 by default) is open on the windows firewall.

4. Save the configuration file with a new name, e.g. c:\sqlcompleteconfig.ini

5. Install the Windows Azure components

6. Create a %WINDIR%\Setup\Scripts\SetupComplete.cmd file. This file is executed once after windows setup completes. It can be used to perform additional unattended setup tasks, including installing applications. It runs under the system authority context. The file should contain the commands to complete the sql installation, such as:

“C:\Program Files\Microsoft SQL Server\100\Setup Bootstrap\SQLServer2008R2\ setup.exe"

/CONFIGURATIONFILE="C:\SQLCOMPLETECONFIG.INI"

/installmediapath="c:\sqlinstall\x64\setup”

7. Sysprep the VM and upload it to Azure as instructed in my previous post.

Remember to configure the appropriate ports & endpoints for the VM when creating the package to upload. Enjoy SQL running on Azure

Azure Connect - a vpn between Azure and your machines

gmarchetti — Tue, 19 Apr 2011 04:18:37 GMT

Azure Connect is a service that lets you establish a vpn tunnel (for want of a better word) between a virtual machine running in Azure and another running on premises. The connection is point-to-point, meaning that you will need to configure it on every machine involved. Nonetheless, it is a valuable tool when you need to access resources within your network from Azure and viceversa.

To set up Azure connect, you will need:

1. An activation token. Open the management portal. If you are on the Connect beta program, you'll see a "connect" icon on the main ribbon. Click on it and then on your subscription name. Select "Get Activation Token" in the top ribbon. Your token will be displayed. Copy it and save it.

2. A local endpoint, i.e. a communication service. To install it on the local machine, click on "Install Local Endpoint" in the top ribbon, copy the link provided (note that it contains the token) and use it to open another browser tab. When asked, click "run" to download and install the endpoint. This is a service running as LocalSystem; it uses the HTTPS protocol, so make sure that TCP port 443 outbound is open.

3. An endpoint group. This is made of local endpoints and azure roles (including VMs) amongst which connectivity is authorized.

Local machines will register with Azure Connect after you install the endpoint.

For Azure roles, you need to modify in Visual Studio:

- The configuration file .cscfg and include the following line under "Settings"

where _token_ is the activation token string that you copied in step 1.

- The service definition file .csdef and include the following line under "Imports"

The same can be done via the GUI. Right-click on the role name, select properties and then "Virtual Network". Select "Activate Windows Azure Connect", then paste the token in the relevant field.

When you publish the project to Azure with Visual Studio, Azure Connect will be activated for those roles.

Virtual Machines are the exception: you have complete control of what the VM contains, no endpoint agent will be installed for you. You must install it in the base image for the vm role before uploading it to Azure. You can obtain the package at

http://waconnect.blob.core.windows.net/client/latest/x64/wacendpointpackagefull.exe

This is not the same URL that you get by clicking "install local endpoint". Also make sure that IPv6 is enabled in your template (it is by default).

You must enable Azure Connect in the configuration files as before for the VM role as well. Installing the endpoint package in the template is not sufficient. If you have a running VM where you forgot to install the agent, you can still log into it and perform the installation manually. Alas, this change won't be persisted to the stored template.

Firewall ports on the local machines and on Azure roles (except VMs) will be configured automatically during the endpoint installation. If you enforce firewall policies, please make sure to:

- allow TCP 443 outbound.

- allow IPv6 protocol, ICMPv6 type 133 and 134 messages (router solicitation and advertisement).

In the end, you will see a list of the activated machines and roles in your management portal.

You can then create a group and specify which local machines and Azure roles are allowed to connect.

The most common use of Azure virtual networks is domain connectivity. To join an azure role to a corporate domain, you must install a local endpoint on a domain controller that is also a DNS server, then include both role and DC in the same endpoint group (other servers can be included too). In the configuration for the role, you must set specific parameters to enable the domain join. You'll find them in the settings for the role.

Powered by Qumana

Azure billing clarification

gmarchetti — Tue, 15 Mar 2011 04:00:33 GMT

A quick clarification: Azure allocates resources to your role as long as it is deployed, whether it is running or not. Billing starts when the deployment is complete and finishes when it is deleted. Roles that are deployed will be billed for the deployed hours, whether they are running or not.

In summary, if you are not using a role, remove it!

Powered by Qumana

Put a VM on Azure

gmarchetti — Sat, 12 Mar 2011 09:57:17 GMT

I have summarized here all the steps you need to take in order to deploy an Azure VM.

Step 1: Get your certificates

I assume that you have an active Azure subscription and you have installed visual studio 2010, the azure sdk and tools and activated the VM role. You will need a management certificate for your subscription to deploy services and 1 or more service certificates to communicate with those securely. To generate a x509 certificate for use with the management API:

1. Open the IIS manager, click on your server.

2. Select "Server Certificates" in the main panel.

3. Click "Create Self-Signed Certificate" in the actions panel

4. Give the certificate a friendly name.

5. Close IIS manager and run certmgr.msc

6. Find your certificate in "Trusted Root Certification Authorities"

7. Right-Click on it, select All Tasks / Export

8. Do not export the private key, choose the DER format, give it a name.

9. Navigate to the Windows Azure management portal.

10. Select Hosted Services / Management Certificates / Add a Certificate

11. Browse to the management certificate file and upload it.

Step 2: Prepare the VM

I assume that you are familiar with Hyper-V and how to build a virtual machine on a hyper-v host.

Create a virtual machine on hyper-v. Note that the maximum size of virtual hard disk you specify will determine what size of Azure VM you will be able to choose. An extra-small machine will mount a vhd up to 15 GB, small one up to 35 and medium or more up to 65 GB. This is just the size of the system VHD. You will still receive local storage, mounted as a separate volume.
Install Windows Server 2008 R2 on the VHD. It is the only supported o/s as of writing.
Install the Azure integration components in the VM. They are contained in the wavmroleic.iso file, which is typically located in c:\progam files\windows azure sdk\<version>\iso. You need to mount that file on the VM and then run the automatic installation process. This provisions the device drivers and management services required by the Azure hypervisor and fabric controller. Note that the setup process asks you for a local administrator password and reboots the VM. The password is encrypted and stored in c:\unattend.xml for future unattended deployment.
Install and configure any application, role or update as you normally would.
Configure the windows firewall within the VM to open the ports that your application requires. It is recommended that you use fixed local ports.
Open and administrator command prompt and run c:\windows\system32\sysprep\sysprep.exe
Select "OOBE", Generalize and Shutdown

This process removes any system-specific data (including the name and SID) from the image, in preparation for re-deployment on Azure. If your application is dependent on those data, you will have to take appropriate measures at startup on Azure (e.g. run a setup script for your application). The VHD is now ready to be uploaded. It is recommended to make a copy of it to keep as a template.

Note that any deployment to Azure starts from this vhd. No status is saved to local disk if the Azure VMs is recycled for any reason.

Step 3: Upload the VM to Azure

For this you will need a command-line utility provided with the Azure SDK.

Open a windows azure command prompt as administrator.
Type

csupload Add-VMImage -Connection "SubscriptionId=<YOUR-SUBSCRIPTION-ID>; CertificateThumbprint=<YOUR-CERTIFICATE-THUMBPRINT>" -Description "<IMAGE DESCRIPTION>" -LiteralPath "<PATH-TO-VHD-FILE>" -Name <IMAGENAME>.vhd -Location <HOSTED-SERVICE-LOCATION> -SkipVerify

The subscription ID can be retrieved from the Azure portal and the certificate thumbprint refers to the management certificate you created and uploaded before. The thumbprint can be retrieved from the portal as well. The description is an arbitrary string, the literal path is the full absolute path on the local disk where you stored your vhd. The image name is the name of the file once stored in Azure and the location is one of those available in the Azure portal. Note that the location must be specific, e.g. "North Central US". A region is not accepted (e.g. Anywhere US). SkipVerify will save you some time.

This command will create a blob in configuration storage and load your vhd file in it for future use, but not create a service or start a VM for you. In the Azure portal the stored virtual machine templates can be found under "VM Images"

Step 4: Prepare the service model

Azure requires a service definition and a service configuration file before deploying any role. These are .xml files that are packaged and uploaded to the fabric controller for interpretation. You can generate one for the VM using Visual Studio 2010.

1. Open Visual Studio 2010 and create a new Windows Azure project.

2. Do NOT add any role to the project from the project setup wizard.

3. In the solution explorer panel, right click on the project name and select New Virtual Machine Role. Note that a service may be made of several roles, including multiple VMs.

4. In the VHD configuration dialog, specify your Azure account credentials and which of the stored virtual machine templates you'd like to use.

5. In the Configuration panel specify how many instances you'd like and what type. Remember the size constraints on the system VHDs.

6. In Endpoints, specify which ports and protocol must be open for your applications within the virtual machine (they should match those configured before).

7. Note that RDP connections are configured elsewhere.

8. Once the VM role configuration is done, right-click on the project name and select Publish. You have an option to create the service configuration package only, to be uploaded later via the portal, or to actually deploy the project. I am assuming that you have not got a service defined yet. It is advisable to configure RDP connections for debugging purposes at least during staging.

9. Select Enable connections, then specify a service certificate. This will contain a private key used to encrypt your credentials. If you have none, you can create one from this interface. If you do create a new certificate, click View, Details and Copy to File to export it. Make sure to include the private key.

10. Specify a user name and password to connect to this virtual machine. Change the account expiration date as necessary (but set it before the certifcate expires).

11. Select "Create Service Package Only" and save the package file.

Step 5. Create the service in Azure

1. In the Azure Management Portal, select Hosted Services / New Service

2. Populate the form, specifying a name for your service and deployment options. Note that the location you select must be the same specified at upload time for the virtual machine you want to use. Select the configuration package and file that you saved before. Add the certificate that you exported before for RDP.

3. Click OK to deploy. Start your deployed machines.

Step 6: Connect and enjoy.

From the machine where you generated the RDP certificate, connect to your virtual machines and test. Simply select the virtual machine in the Azure portal and click "connect". A RDP file will be generated for you to save and open. Once debugging is finished, it is recommended to disable RDP connections for production.

Powered by Qumana

The Hyper-V Cloud - no clusters?

gmarchetti — Thu, 25 Nov 2010 02:19:02 GMT

Microsoft has recently published a set of guides to build your own private cloud solution using Hyper-V, System Center Virtual Machine Manager and its Self-Service Portal 2.0

They cover planning, deployment and operations. You can find them here.

Note however that the guides assume a deployment on stand-alone servers. There is no discussion of clustering for high availability.

I'd like to add a few observations derived from experience of running a private cloud implementation that includes clusters too.

Storage

It seems obvious, but local storage is not relevant for clusters. SCVMM will explicitly place virtual machines ONLY on cluster volumes (dedicated or shared). You can still manually create virtual machines on directly-attached disks, but it just complicates things, as they are not easily distinguishable from highly available ones in VMM. It makes sense to purchase systems with a hardware-mirrored boot volume only, plus whatever dedicated storage adapter is appropriate to your workload.

Clusters tend to drive high consolidation ratios to the storage, for the simple fact that it is shared amongst several nodes. The number of sustained IOps often becomes more relevant than the maximum theoretical bandwidth, as during VM operations you may often find that a lot of relatively small I/Os are performed with a random access pattern. Fibre channel may fit such a profile better than iSCSI. Ideally, you'll profile the load before consolidation, but in a private cloud / infrastructure-on-demand environment you may not have the luxury.

You'll be well advised to consult the storage vendor's planning guide beforehand.

Storage vendors often publish the IOps and bandwidth ratings of their arrays. For instance, for the HP P2000 G3, you will find that 10 Gb/s iSCSI (with dedicated adapters) and 8 Gb/s FC are comparable in bandwidth utilization for large-block sequential reads and writes. However, FC still sustains 20% more IOps than iSCSI with a random 60/40 mix of read-write operations of 8KB blocks.

Interestingly, 6Gb SAS is equal or slightly better than FC in HP's measurements, which used 4 directly-attached servers (no fabric) for testing. Results may vary when a fabric is involved.

Servers

Blade servers have grown in popularity and are often recommended for private cloud solution, thanks to their good price / performance ratio, density and flexibility. However, in a highly available implementation due consideration must be given to:

- Connectivity: Microsoft recommends at least 4 network ports + 1 storage port (2 better) for each node in a hyper-v cluster. Blade connectivity may limit your options.

- I/O performance: i/o bandwidth and operations per second depend on the midplane capabilities. Chassis capable of full redundancy with non-blocking backplanes and 10 Gb/s per lane are available - at a price (e.g. HP c7000). Cheap solutions typically involve some element of oversubscription or no redundancy.

- Reliability of the shared components. For instance, in a study published on the IBM Systems Journal, the lowest mean time to failure (MTTF) belonged to the chassis switch modules, followed by the blade base board. An equivalent external Cisco switch lasts about twice longer between failures, according to published specifications. Even so, according to the same study, a blade availability can reach 99.99% and it is possible to build a 99.999% available infrastructure with blades by having at least 1 "hot-standby" in the chassis, in addition to using redundant components where possible.

High-end servers (e.g. IBM x3850) are also interesting virtualization platforms, because of hardware becoming more affordable and Windows Datacenter unlimited virtualization rights. They drive high consolidation and operating efficiency, mainly thanks to capacity up to 1TB RAM, 64 cores. They enhance the availability of the basic platform by providing features not commonly found elsewhere, like:

- ChipKill or SDDC memory with hot add / replace.

- Hot-add / remove of i/o adapters.

- Hot-add CPUs.

Particular attention must be given to the memory configuration of the servers, especially with the recent NUMA chipsets from Intel and AMD:

- Populate processor local memory banks with equal amounts of RAM.

- Populate memory controllers with equal capacity.

- Populate all memory channels for each controller evenly to exploit the maximum memory bandwidth.

- Use dual-rank DIMMs.

Network

Whilst hyper-v will work with any supported network adapters, it is important to notice that certain features will be available only with the appropriate combination of chipsets.

VMQs (hardware-managed network queues for VMs) are highly recommended for best performance and consolidation ratios, but they require support for interrupt coalescing (in R2), the latest Intel Pro or Broadcom chipsets, appropriate drivers and some registry hacking, as explained here and here.

VM Chimney (TCP offload for VMs) has proven unreliable due to driver issues in my experience. I'd rather have VMQs.

Note that if you enable IPSec or any filter driver on a particular connection (e.g. Windows firewall), that connection may not be offloaded.

Microsoft does NOT support network teaming with Hyper-V. Teaming is supported by the OEMs. VMQs and teaming may be mutually exclusive, depending on vendor.

Operating System Editions

In order to build fail-over clusters, you will need enterprise or datacenter editions. Note that Hyper-V Server R2 is also capable of clustering and similar in many respects to enterprise edition. The smaller footprint of Hyper-V Server R2 implies the need to patch it less often than a full edition, hence it is ideal to minimize planned downtime.

It is possible and legal to purchase datacenter edition (for the unlimited licensing) but deploy Hyper-V Server R2 (for the reduced footprint), transferring the licenses.

Cluster Size

There are several considerations to determine the number of nodes and machine per node in a cluster:

- The officially supported maximum is 64 VMs per node in a hyper-v R2 cluster (increasing to 384 in SP1).

- The officially supported max virtual / physical core ratio is 8:1.

- Large clusters are more likely to incur in the WMI issues mentioned in my previous post.

- It is fine to have the CPU capacity, how about the i/o? How many iops per node can you sustain with your adapter / SAN combination?

Cluster Shared Volumes

Assuming that you want live migration to minimize planned downtime and optimize allocation of resources, you will need CSVs. A common question is how many machines to deploy on each CSVand how many CSVs to have per cluster. The answer to that depends on several factors. In my experience, the most troublesome one is the data you need to back up and how long you can tolerate reduced performance during the backup.

Each time you perform a CSV backup, the server hosting the VM to back up requires ownership of the whole CSV volume to snapshot. I/O to the volume is redirected over the network for all VMs hosted on that volume, with consequent performance impact. The time you can tolerate that, multiplied by the backup throughput will give you the max amount of data to put on that CSV. Divide that by the average size of a VHD and you'll have a rough estimate of how many VMs will fit.

Rules of Thumb

I claim no scientific basis for the following rules, other than my empirical observations. Here they go, in no particular order:

1. Keep the number of nodes in a cluster small (2-4) to avoid annoying SCVMM bugs.

2. Assume a random 60/40 i/o pattern if you don't know in advance what your VM workload will be. It is quite common, in my experience.

3. Plan for at least 1 management network, 1 for heartbeat, 1 for live migration, 1 dedicated to VMs and 1 dedicated storage adapter. For higher consolidation ratios and availability, plan on 2 storage adapters with MPIO.

4. Use separate VLANs for management and VMs to isolate traffic and for ease of administration. Consider 10GbE for the shared VM networks in order to minimize the number of adapters and cables.

5. The quality of VSS (snapshot) and VDS (disk) providers varies greatly with the OEM. Be sure to test them. A snapshot should NOT take longer than 10 seconds or it will fail.

6. If you don't know how long you can tolerate redirected i/o, 30 minutes is a useful maximum. I have seen CSVs crash in several occasions (again, depending on OEM) after that.

7. Fail-over clusters do NOT take into account connectivity to the VMs. In other words, if you are sharing the VM network with the host o/s and for some reason that connection fails, the cluster may fail over to another node its own IP addresses on that network but not the VMs attached to it.

8. A few large servers are easier to manage than many small blades, if you implement appropriate procedures to minimize downtime and have a support contract when things go wrong to fix them quickly :-) They may also be more cost-effective, if you drive consolidation and take advantage of power optimization technology.

9. Use group policies to control patching with WSUS or similar. Do NOT use the default "download and install at 3am" option on all cluster nodes, or they will all reboot at the same time.

10. If you don't know your storage vendor's iops ratings, use these ballpark figures on wikipedia.

11. On your hosts, make sure that the antivirus excludes .vhd, .vsv, .avhd files and vmms.exe, vmwp.exe. If you are running hyper-v server only, do you need an antivirus on the host? This is not a rhetorical question by the way; I am interested in opinions.

12. If you don't know the size of your CSVs in advance, 2 TB works on all MBR and GPT disks. Most backup and restore utilities, snapshot providers etc... can handle 2 TB. It is also a tolerable size should you ever need to run chkdsk or defrag on the volume (few, large vhd files should not cause much trouble or take much time to fix in that respect).

13. Both fail-over and PRO have no idea of virtual applications, i.e. applications that require a set of interconnected virtual machines. They may move them on different nodes. A way around it is to script the appropriate migration sequence with Powershell.

Powered by Qumana

Patches and KB articles for Hyper-v R2

gmarchetti — Sat, 20 Nov 2010 01:25:21 GMT

I'd like to share with you some tips to mitigate a few frustrating intermittent problems with hyper-v and system center.

1. Data Protection Manager 2010

BEFORE installing DPM agents on systems running Hyper-V R2, you must install hotfixes KB975921 and KB975354. Note that if you had a beta or rc version of dpm, upgrading the dpm server will NOT upgrade the agents and you will not be prompted to do so. You will only see "inconsistent replicas".

The first KB refers to a situation where the volume snapshot provider for your clustered volume crashes and does not return a "completed" signal. Alas, the volume is not returned to its original state, the cluster resource fails and all its virtual machines with it. You may be lucky and never encounter this problem; alas the quality of snapshot providers varies widely so I suggest you install the patch anyway.

The second KB refers to a common scenario where you want to backup virtual machines running on different nodes of a cluster at the same time. If the machines sit on the same cluster volume, ownership of the volume must be transferred to the node requesting the snapshot at that point. Alas this transfer may happen before the post-snapshot steps of a virtual machine backup are complete, so the replica is inconsistent. If at all possible, I suggest to build the DPM protection groups and time the backups in such a way that frequent transfers of volumes are avoided.

2. Virtual Machine Manager and Operations Manager

VMM and OM agents rely on WMI heavily. When you consolidate dozens of virtual machines on a cluster and want to take advantage of the PRO functionality (hence run both agents on all nodes), the WMI service is heavily loaded and may crash. It restarts without too much fuss and no data is lost, but anything depending on it fails. In your VMM console you may notice that all machines on a node seem to fail at the same time, or that the connection to the VMM agent running on that node times out. Operations manager will also issue critical alerts. Live migrations and deployments may be interrupted. To mitigate the problem, you must install KB974930 and KB981314.

The first KB addresses a memory leak of the Win32_Service WMI class. The second KB addresses timeouts in WMI queries about a failover cluster object.

You may also want to increase the number of concurrent connections and the timeout period for the Windows Remote Management service (winRM). To do so, you could run this script on all nodes of the cluster:

winrm set winrm/config/Service @{MaxConcurrentOperationsPerUser="400"}

winrm set winrm/config @{MaxTimeoutms = "1800000"}

net stop vmmagent

net stop winrm

net start winrm

net start vmmagent

The script increases the number of concurrent WM operations to 400 (the default is 200) and sets the default timeout to 30 minutes (it should be plenty). Note that this is just the default timeout - operations may specify their own.

For those values to be taken into consideration, the script then restarts the vmmagent and winrm services.

In my experience applying these fixes reduces the frequency and duration of such timeouts, but does not eliminate them completely. You may have to tweak the numbers over time to improve the situation further.