John Howard - Senior Program Manager in the Hyper-V team at Microsoft

NIC Teaming in Windows Server "8" Beta whitepaper now available

John Howard -MSFT — Thu, 05 Apr 2012 21:42:12 GMT

For those of you interested in learning more about the new inbox capability for NIC Teaming (aka LBFO or Load Balancing/Failover) in Windows Server "8" Beta, there's an excellent whitepaper which has just been published. It covers pretty much everything you need to know from the feature overview, the different modes it can operate in, load distribution algorithms and considerations, the user interface, PowerShell cmdlets to configure it, how to use it with Hyper-V (and SR-IOV), plus more.

Here's the download link.

Cheers
John.

Everything you wanted to know about SR-IOV in Hyper-V Part 8

John Howard -MSFT — Wed, 21 Mar 2012 17:00:41 GMT

This part of the series is all about determining why SR-IOV may not be operational. As you will discover, there are several reasons, some of them obvious if you’ve followed all the parts so far, some more subtle. By the end of this part, you will be an expert!

Assuming you have a switch in SR-IOV mode, and have enabled SR-IOV on a virtual network adapter, the most obvious place you will notice that SR-IOV isn’t working is in Hyper-V Manager after selecting the networking tab for a running virtual machine. (I love this panel – my favourite bit of Hyper-V Manager that I worked on for the Windows “8” release!)

I’ve already outlined dependencies from past posts. But let’s assume you haven’t heeded them and have done this on an older machine which isn’t SLAT capable, doesn’t have BIOS support for SR-IOV and doesn’t even have an SR-IOV capable network adapter, as for the following screenshot. The first clues will come from the Get-VMHost PowerShell cmdlet. In this case, IovSupportReasons property returned from the cmdlet is pretty verbose in outlining a number of issues.

Essentially you’re never going to get SR-IOV working on the above machine. So let’s move on…

The follow example is a machine which has chipset support, but the BIOS doesnt have support for SR-IOV. This is probably the most common error you will find on servers currently shipping, or if you were to install Windows Server “8” beta on a desktop class machine. The error specifically is the first entry which says “To use this SR-IOV on this system, the system BIOS must be updated to allow Windows to control PCI Express. Contact your system manufacturer for an update.”

Next, let’s assume that the machine has chipset support, the BIOS has SR-IOV support, and you’re using a NIC which is capable of SR-IOV, but it still isn’t working. In this case, Get-VMHost may return the following:

In addition, after a virtual network adapter is started (by changing the state of a virtual machine to running, or by toggling the IOVWeight property on a running virtual network adapter to a positive value in the range 1..100) the following may be logged in the event log indicating that the user of SR-IOV has been disabled by policy on this system.

This reason for this takes a little explaining. Even if the system manufacturer has made the necessary changes in their BIOS for the base functionality Windows requires to support SR-IOV, some chipset implementations have flaws in them. In some cases, system manufacturers may be able to work around the problem by a fix in firmware. This is not universally true, and it may be a case that it requires a revision to silicon that cannot be fixed by firmware alone (in other words, a revised motherboard). The result of the chipset flaws are such that it is possible for a guest operating system which has a VF assigned to cause the physical system to operate with reduced performance, or in the worst case cause it to crash.

If you are prepared to assign VFs only to “trusted” workloads in lieu of an updated BIOS with a workaround (assuming it is possible on your hardware), the following registry key can be added on the parent partition. IOVEnableOverride. Type DWORD. Value 1. Under HKLM\Software\Microsoft\Windows NT\CurrentVersion\Virtualization. The system should also be restarted after setting this key. (Technically you could restart the VMMS service and save/restore each running VM which has an IOVWeight set as well.)

On a restart, the following event will be logged on each startup. As long as you are comfortable and understand the potential risk involved, SR-IOV should now work on a system with this registry key set.

If your system manufacturer can work around the chipset flaw, and has provided a BIOS which incorporates a workaround, the registry key is not required, the event above will not be logged, and VFs can be securely assigned to virtual machine. In these cases, if a virtual machine with a virtual function assigned can trigger the conditions which would otherwise cause the symptoms previously described, Hyper-V will automatically remove the VF from the VM and let it continue running using software based networking. However, it should be noted that if there is a VM which is able to trigger one of the conditions, there is an extremely likely probability that the guest operating system is compromised and likely to crash very soon after. However, the remainder of the system including other running VMs will not be affected.

The next useful cmdlet is Get-NetAdapterSriov. This cmdlet gives a lot of useful information about the physical network adapter, assuming it supports SR-IOV.

It’s pretty telling that nothing was returned. A clear indication that there are no SR-IOV capable network adapters. Let’s instead run this on a machine which does have an SR-IOV capable network adapter.

The fact that something was returned indicates the network adapter is SR-IOV capable. Furthermore, looking at NumVFs, we can see that this adapter is working correctly and has available resources.

If you’ve created a virtual switch, the third useful cmdlet is Get-VMSwitch. Remember that to enable SR-IOV, the switch must be created in SR-IOV mode to start with. When SR-IOV is not available on the physical NIC, there are a number of properties which indicate why. IovVirtualFunctionCount and IovQueuePairCount will be zero. IovSupport will be false, and IovSupportReasons will list the reasons why.

First an example where the machine itself does not support SR-IOV, and the switch is bound to a network adapter which doesn’t support SR-IOV either.

Here’s an example where the machine does support SR-IOV, but the physical network adapter does not. IovSupportReasons is clear as to the cause of the problem, regardless of whether the virtual switch is created with SR-IOV enabled or not.

And another example where the machine supports SR-IOV, as does the physical network adapter, but the switch was not created in SR-IOV mode. This one is a bit more subtle to spot as IovSupport and IovSupportReasons indicate everything is OK. The property IovEnabled is False, hence IovVirtualFunctionCount is zero even though the physical NIC has resources potentially available.

On a “good” (well configured) machine, you will get very different results in these properties. Notice how there is a positive integer in IovVirtualFunctionCount, IovSupport is True, and IovSupportReasons has a single value in the array, “OK”.

The last cmdlet is Get-VMNetworkAdapter. This should be run against a running VMs network adapter. Here again is an example from a physical machine which does not support SR-IOV, and does not have an SR-IOV capable network adapter. Even though the IovWeight property is non-zero, note that IovQueuePairsAssigned and IovUsage are zero, and Status and StatusDescription contain a slew of reasons why the network adapter is degraded.

Here’s the same on a “good” machine for comparison. Notice that IovUsage is 1.

The above has covered the common cases, but there are slightly more subtle ones, those around when port policies have been applied. See if you can spot what’s wrong in the following output. In this case, the machine is fully capable of SR-IOV, the virtual switch is in SR-IOV mode, and the IovWeight has been set on the network adapter correctly. It’s none of the reasons described so far.

Unfortunately, the StatusDescription isn’t overly helpful in indicating the precise reason. In fact, for several technical reasons, this is something which is incredibly difficult to accurately provide, so is unlikely to change before final release. Instead, we need to look at the policies which have been applied. In this particular case, I enabled RouterGuard on the VM. When we apply policy which can only be enforced by the virtual switch, and not the physical NIC, we automatically disable the use of SR-IOV on the VM so that the policy can be applied. Turning off any such policies (assuming they are compatible with the networking configuration requirements of the VM) will enable SR-IOV to start operating again.

Now I did mention it in an earlier post, but if you are still struggling to get SR-IOV enabled and you believe you have everything you should need (chipset, latest BIOS, BIOS settings, NIC, virtual switch in SR-IOV mode), there is one other thing that is definitely worth checking. Some BIOS’s have more than one firmware setting to enable SR-IOV. If in doubt, always go back to your system manufacturers documentation to make sure you have the settings configured correctly. And remember, if you do change BIOS settings, you may need to hard power cycle the machine, not just a soft restart.

There are two other reasons worth mentioning. One is if you are using client Hyper-V. As this is a server only feature, the user interface for SR-IOV does not exist in Hyper-V Manager on client. (Note that the SR-IOV options will appear though if you are using Hyper-V Manager on a client connecting to a remote Windows “8” server.)

If you were to run get-vmhost on a client, it will indicate that SR-IOV is not supported.

And similarly for a virtual switch (sadly my laptop doesn’t have a 10G network adapter that supports SR-IOV either – next upgrade )

So that’s pretty much it in terms of diagnosing why SR-IOV may not be operating. If you understood all the above, you are now a fully-fledged superhero and have earned your cape with honours!

Probably one more part to come in this series, the “kitchen sink” part, as in everything not already mentioned. That will hopefully be early next week after I find time to write it.

Cheers,
John.

Everything you wanted to know about SR-IOV in Hyper-V Part 7

John Howard -MSFT — Tue, 20 Mar 2012 19:02:00 GMT

Only a few more topics in this series remain uncovered. This part of the series looks at resiliency with SR-IOV. By resiliency, I mean, of course, NIC teaming, otherwise known as Load Balancing/Failover (LBFO), the ability to aggregate network links or fail over should a link fail. I won’t get into the details of configuring NIC teaming which is inbox in Windows Server “8”, just how it relates to the use of SR-IOV.

When a NIC team is created on top of two or more SR-IOV capable physical network adapters, the SR-IOV capability is not propagated upwards. Hence, the two features are not compatible in the parent partition.

The solution for virtual machine networking redundancy with SR-IOV and Windows Server “8” guest operating systems, is to do teaming inside the virtual machine guest operating system itself. To configure this:

Each physical NIC should have a virtual switch bound to it in the parent partition, each with SR-IOV enabled
The VM is configured with two network adapters, each connected to one of the virtual switches.
In the parent partition, the virtual machines network adapters MUST be configured to allow teaming (meaning each network adapter can spoof the MAC address of the other) by running the following PowerShell command:

Get-VMNetworkAdapter –VMName “VMName” | Set-VMNetworkAdapter –AllowTeaming On

Configure the IOVWeight on the virtual network adapters as covered in a previous part of this series
Configure teaming in the guest operating system in switch independent, address hash distribution mode.

The following is a screenshot from inside a virtual machine configured in this way, with one of the physical links failed (by disabling one of the NICs using ncpa.cpl in the parent partition).

It is possible to create a team in the guest operating system where, in the parent partition, one virtual switch is in SR-IOV mode and the other not (or bound to a network adapter which does not support SR-IOV). This is fully supported, but you should bear in mind that it may introduce side effects. LBFO is not aware of what is backing the NICs in the team, hence not aware that one path is SR-IOV enabled (potentially with a VF), and the other path not. In this situation, while a VM still has redundancy against link failure, you may want to configure the virtual NICs for Active/Standby operation inside the VM.

While a team interface can be created on up to 32 NICs, inside a virtual machine, the (unenforced) support limit is for 2 NICs.

Note that the networking tab in Hyper-V will not show the IP address(es) of the virtual network adapters when a team has been created inside a VM. This is because the virtual network adapters do not have an IP address. It’s the team interface inside the VM which has an IP address.

The following screenshot was taken with the physical NIC bound to virtual switch “TeamDemoB” disabled.

Hyper-V Manager is correctly showing the status of the second virtual network adapter as “OK” from the Hyper-V perspective. This is because it still has connectivity to other virtual machines connected to “TeamDemoB”. In other words, Hyper-V Manager does not ripple through the underlying physical link status.

In the next part of this series, I’ll give you the tools needed to make you an SR-IOV debugging superhero.

Cheers,
John.

Everything you wanted to know about SR-IOV in Hyper-V Part 6

John Howard -MSFT — Mon, 19 Mar 2012 13:51:00 GMT

Summarising the series so far, we’ve answered “Why SR-IOV?”, looked at hardware and firmware dependencies, and run through the user interface and PowerShell cmdlets to configure SR-IOV. Next on the agenda is how SR-IOV and Live Migration co-operate. In addition, there’s a video showing covering the configuration aspects of SR-IOV from the previous part which also shows Live Migration in action.

A goal very early on in Windows “8” planning was that we should consider features which are incompatible with mobility scenarios such as Live Migration. (Actually SR-IOV has been in the pipeline for considerably longer than that as you can probably tell from the deck back at WinHEC 2008 I linked to previously, and even talked about at WinHEC 2006.)

That goal poses somewhat of a problem when hardware is assigned to a virtual machine. Let’s ignore SR-IOV for a moment and take a step back a few years into the initial development and prototyping of the feature. You may be familiar with the term “Discrete Device Assignment”. This is where a fully-fledged PCI Express device is assigned to a virtual machine. Discrete assignment, from a software engineering perspective can be viewed in some ways as a stepping stone towards SR-IOV support. However, discrete assignment is fraught with issues in several areas:

Security
Usability & Mobility
Scalability

From a security perspective, a virtual machine with significant uncontrolled access to hardware is too risky to entertain in a shipping product, while still providing a production support statement. It would be extremely difficult to secure the VM to the extent that it was unable to cause side effects to other partitions, thus breaking a critical security tenet.

From a scalability perspective, it is difficult to describe assigning a device taking an entire PCI Express slot into a single virtual machine as scalable. It would be very expensive to scale to tens or hundreds of VMs in this manner. If indeed, you could find a server with that many PCI Express slots.

From a usability perspective, it is difficult to describe this as a common user scenario. Granted, there are niche exceptions which do have some merit, but several things would be lost in the process. The most important of these is Live Migration support, shortly followed by state changes (apart from running or shutdown), then snapshots. Probably more as well. The reason is that is extremely difficult to save the state of a piece of arbitrary hardware inside a running VM, and subsequently restore it to a running state on a different platform. Further, even if we could temporarily halt the VM with hardware state intact (as would be necessary during Live Migration black-out), without an absolutely identical configuration on the target at all levels (not just hardware) the chances of successful restoration are non-existent.

Hence, we considered discrete assignment not very useful except to an extremely niche segment of our user base who aren’t concerned about security, scalability or mobility. Instead, we focused on something that addresses all of these concerns, namely SR-IOV. Security is built in at all levels. Only a single PCI Express slot is needed to support many virtual machines. And we support mobility (live and quick migration), state changes and snapshots.

So you may be wondering how SR-IOV overcomes the statement I made about it “being difficult to save the state of a piece of hardware inside a running VM, and subsequently restore it to a running state on a different platform”, yet still be able to achieve all these goals. After all, a VF is true hardware and it running in the VM.

As simple as it is, the answer probably isn’t immediately obvious. The answer is that we don’t save the hardware state at all, and don’t even attempt to tackle the problem. And yet we are able to migrate to a platform which could have a completely different physical NIC, the same type of NIC at a different firmware release level, or even a platform which doesn’t have SR-IOV support. And through all of these scenarios, keep networking fully functional in the VM. Confused yet?

One more minor backtrack. You may have noticed I’ve said a couple of times that a VF “backs” a software based network adapter. By this I mean that the VM always has a software based network adapter, but when a VF is available, we “failover” automatically to the hardware path for I/O. The software path is always present, only the VF is transient. So now it should be a little more obvious. Whenever we go through a state transition that would require hardware state to be saved, we remove any VFs from the VM beforehand, falling back to software based networking. (I say VFs in plural as a VM can have multiple software network adapters, up to eight, hence up to eight VFs assigned.) Once the VF is removed, we can perform any operation necessary on the VM as it is a complete software based container at that point. Once the operation has been completed, assuming hardware resources are available and other dependencies met, we will give a VF back to the VM. This completely solves the problem.

Those paying attention may have noticed I said that “we remove any VFs from the VM beforehand” and are wondering whether there are implications if the guest operating system isn’t co-operative. The short answer is no, our state model covers this scenario, although it is certainly easier when the guest operating system co-operates in VF deallocation.

Note also I used the word “failover” in quotation marks. I could have said “team”, but that implies a little more functionality than “failover”. Truthfully, we haven’t come up with a good term yet, but the point is that this “failover” is nothing to do with NIC teaming, also now native in Windows Server “8”. It simply means that we use a VF automatically if it is present, or software based networking if it is not, and during the transition either way, there will be no loss of network packets. As we will see shortly, NIC teaming and SR-IOV can co-exist as well in a virtual machine.

At this point, the adage about a picture speaking a thousand is apt. Rather than attempt a series of badly drawn block diagrams, here’s a video showing SR-IOV configuration and Live Migration. There are so many other new features in Hyper-V also shown in this video which aren’t immediately obvious, I’m struggling to contain myself, but do manage to limit myself at least to talking just about SR-IOV in the recording. Maybe you will notice the use of an SMB file share for the VM, the use of VHDX, Live Migration without a cluster, and of course PowerShell support.

Windows Server “8” Beta. Demonstration of SR-IOV in Hyper-V and Live Migration

More to follow in the next part.

Cheers,
John.

Everything you wanted to know about SR-IOV in Hyper-V Part 5

John Howard -MSFT — Fri, 16 Mar 2012 16:15:00 GMT

In parts 1 through 4, I covered the external dependencies and the “why” of SR-IOV. So it’s about time I showed you how to setup SR-IOV and what it looks like in a little more detail from a configuration perspective, both through the user interface in Hyper-V Manager, and from PowerShell.

In part one, I showed a block diagram of how out the indirect I/O model works for virtual machine networking. Here’s a similar block-diagram showing, at an extremely high level, how this changes with SR-IOV. For simplicity, I am showing a single VM with a single VF.

A few key points I want to bring out from this diagram:

I’m not an artist!
The Virtual Switch in the parent partition is in “SR-IOV" mode”
The I/O data path from the VF does not go across VMBus or through the Windows Hypervisor. It is a direct hardware path from the VF in the VM to the NIC
The control path for the VF is through VMBus (back to the PF driver in the parent partition)

The first step when allowing a virtual machine to have connectivity to a physical network is to create an external virtual switch using Virtual Switch Manager in Hyper-V Manager. The additional step that is necessary when using SR-IOV is to ensure the checkbox is checked when the virtual switch is being created. It is not possible to change a “non SR-IOV mode” external virtual switch into an “SR-IOV mode” switch. The choice must be made a switch creation time.

This can also be done through PowerShell using New-VMSwitch. New-VMSwitch requires a parameter to specify the physical network adapter which is going to be used. The physical network adapters can be identified using Get-NetAdapter. In the following screenshot, I have a machine which has multiple physical NICs, one which is an onboard NIC, not capable of SR-IOV, and two dual-port PCI Express 10G NICs which are capable of supporting SR-IOV. Note that I have given some of the adapters “friendly names” using the network control panel (ncpa.cpl).

The following two screenshots show the different ways to use New-VMSwitch to create a virtual switch bound to SR-IOV capable network adapters from the previous screenshot. Note the use of the -EnableIov parameter

Let’s look at the properties we expose on the VMNetworkAdapter object in more detail.

IovEnabled: True if the switch is created in SR-IOV mode, False otherwise

IovVirtualFunctionCount: The number of VFs that are available for use by virtual machines. This will vary by vendor. Note that each software based NIC can be backed by a VF, and each VM can have up to 8 software based NICs.

IovVirtualFunctionsInUse: The number of VFs currently being used by running VMs. In the screenshot, the number is 1 as I have a single running VM with a single software based NIC in SR-IOV mode.

IovQueuePairCount: The number of queue pairs available as hardware resources on the physical NIC. This will vary by vendor. There will be as many queue pairs available as there are VFs, although some vendors may have more queue pairs available than there are VFs. I recommend you generally think of a VF as the entity being assigned to a virtual machines network adapter rather than one or more queue pairs. However, a VF requires at least one queue pair to operate. If the NIC vendor supports additional features such as RSS in a VM backed by a VF, more than one queue pair may be required for a VF. For more information, you should consult NIC vendor guidance.

IovQueuePairsInUse: The number of hardware queue pairs currently allocated to VFs assigned to running VMs.

IovSupport/IovSupportReasons: Array of numeric codes and descriptions regarding the status of the network adapter. More information on these properties will be covered in the “debugging why SR-IOV doesn’t work” part of this series.

Once a virtual switch has been created, the next step is to configure a virtual machine. SR-IOV in Windows Server “8” is supported on x64 editions of Windows “8” as a guest operating system (as in Windows “8” Server, and Windows “8” client x64, but not x86 client). We have rearranged the settings for a virtual machine to introduce sub-nodes under a network adapter, one of which is the hardware acceleration node. At the bottom is a checkbox to enable SR-IOV.

Under the covers, this checkbox is setting a property, IovWeight. This is identical in functionality to VMQWeight in Windows Server 2008 R2, and expresses a desire for a hardware offload, not a guarantee. A positive number between 1 and 100 is “on”, and 0 is “off”. We do not, in Windows Server “8” use a relative weighting system. All numbers between 1 and 100 mean the same. This design allows us to add ‘weighting’ functionality in the future without needing to change APIs.

As for switch creation, enabling SR-IOV on a virtual machines virtual network adapter can be done through Powershell using Set-VMNetworkAdapter by setting the IovWeight property as per the following screenshot.

Assuming you have all the requirements met for SR-IOV, you will see the status change on the networking tab in Hyper-V Manager for a selected VM to “OK (SR-IOV active)”

Let’s go back to that previous PowerShell output and examine the SR-IOV related properties of the VMNetworkAdapter object.

IovWeight: Discussed above

IovQueuePairsRequested/IovQueuePairsAssigned: These are for advanced networking features for a VF. One example is for RSS in a virtual machine (when backed by a VF), and requires that the physical network adapter itself supports RSS on a VF. Note that this is the first time we have been able to achieve RSS in a VM. While this series of posts isn’t about RSS, its benefits, or how to configure it, it’s worth a little diversion. More information about RSS, first introduced in Windows Server 2008, can be found here.

By default, the IovQueuePairsRequested will be set to 1, and it can never be less than 1. If the VF hardware supports RSS and you have a multi-processor VM, you can use this parameter to request additional queue pairs from the set of hardware resources available to allow the VM to scale. It is, however, a request, and the actual number of queue pairs assigned may be less, depending on hardware resources. The number assigned will be in IovQueuePairsAssigned.

IovInterruptModeration: Many modern physical NICs have an advanced property to allow the driver to be able to moderate interrupts. As there are now multiple functions (PF and VFs) which process interrupts, this property allows the VF driver to be able to adapt depending on load. The underlying implemented is up to the driver writer. Hence you should refer to the NIC vendor for guidance as to whether this is implemented, or what the recommended setting should be according to workloads running. The possible values are Default; Adaptive; Off; Low; Medium and High. In most cases keeping the default of “Default” will be sufficient.

IovUsage: Will have value 1 if a VF is actively being used by a VM, 0 otherwise.

Status/StatusDescription: Array of numeric codes and descriptions regarding the status of the network adapter. These are not exclusive to SR-IOV, although we do populate them when IovWeight is set but not working correctly. More information on these properties will be covered in the “debugging why SR-IOV doesn’t work” part of this series.

VirtualFunction: This provides a lot more information about the VF itself, but to all intents and purposes, you can ignore this property. However, it could be potentially useful to scripters to be able to tie back to the physical interface being used on the system through the ifAlias and ifDesc properties. For those who really want to know the full gory details of this object, here’s the full output:

VirtualFunctionsAssigned: This will be deprecated before final release and can be ignored in Windows Server “8” Beta. IovUsage is the parameter to use.

So that pretty much covers the user interface and PowerShell side of SR-IOV configuration. In the next part, I’ll cover Live Migration and show SR-IOV in action with a short video.

Cheers,
John.

Everything you wanted to know about SR-IOV in Hyper-V Part 4

John Howard -MSFT — Thu, 15 Mar 2012 15:07:00 GMT

So far in this series, we have discussed the “why” question, and identified three dependencies for SR-IOV in Hyper-V in Windows Server “8” beta:

System hardware support in the form of an IOMMU device
A PCI Express networking device which has SR-IOV capabilities
A driver model to support both PF and VFs.

This part of the series adds one more major component to that list, and one minor one. The minor one is that the system must support SLAT (Second Level Address Translation). In the server ecosystem, you would be hard pushed to find a system which has an IOMMU chipset which didn’t have a SLAT capable processor, so to almost all intents and purposes this is a non-issue.

The major component missing so far is system firmware, or BIOS. While we have a multi-page document shared with the hardware ecosystem firmware developers describing the necessary changes (and other minor dependencies), the details of that document are beyond what is necessary to understand SR-IOV from an end-user perspective. What you do need to understand though is that without those changes, Hyper-V cannot enable SR-IOV, even if we have chipset support in the form of an IOMMU device actually present.

There are a couple of things from the necessary firmware changes which permeate into the end user usage of Hyper-V which I will cover in more depth in a “debugging why SR-IOV isn’t working” part of this series. For now I’ll just say that one is PCI Express Native Control being transferred to the operating system, and the other is support for a chipset issue workaround.

We have been working with system vendors for a long time to make sure that systems are available for you to try out SR-IOV on in Windows Server “8”. We are on the cusp of the next generation of release cycles from system vendors over coming months, and there will be much wider support in those latest generation platforms. However, for now, the numbers of systems which can be used are relatively limited. Certainly do not expect SR-IOV to work on desktop platforms (or laptops) due to the firmware requirements. This is very firmly (pun intended) a server based feature. For up to date details, you should contact your system manufacturer for information for BIOS versions and platform support on existing hardware. There are supported platforms available today from Dell, Fujitsu and HP, and other OEMs have systems in “best effort” support. Some of this information is listed in the release notes for Windows Server “8” beta.

One thing I would point out though is that some BIOSs we have seen have more than one place where SR-IOV must be enabled. So please do take the time to make sure you follow your system vendors instructions accurately.

An interesting presentation 4 years ago at WinHEC 2008 was made by my colleague, Jake Oshins. It has some further references if you are interested in further reading about some of what has been covered so far in this series of posts. Some information has changed, but most still applies.

So now that we have enumerated all the hardware and software components needed for SR-IOV support which are (largely) outside the control of Microsoft, the next part of this series is where we’ll start to see SR-IOV in action.

Cheers,
John.

Everything you wanted to know about SR-IOV in Hyper-V Part 3

John Howard -MSFT — Wed, 14 Mar 2012 15:42:36 GMT

So far in this series, we’ve looked at the “why” question, and the hardware aspects of SR-IOV, and identified that to use SR-IOV in Hyper-V it is necessary to have system hardware support in the form of an IOMMU device, and a PCI Express device which has SR-IOV capabilities. Now it’s time to start taking a look at the software aspects. In this part, I’ll look at device drivers.

SR-IOV Device Drivers

I’ve mentioned that a driver model is essential for a total SR-IOV solution. Yet the SR-IOV specifications don’t mention anything about driver models. Hence it has been up to Microsoft to specify the necessary interfaces necessary to support VFs and PFs running together in a virtualised environment. You may be asking why we need to update the driver model for SR-IOV when we already have a well-established driver model for networking, in the form of NDIS. If you think about a “traditional” networking device driver, the driver is only responsible for controlling a single device (in the parent partition, or the only instance of Windows when outside of virtualization). That device, of course, being what in the SR-IOV world is referred to as the PF.

Prior to Windows “8”, we’ve only had emulated or software based devices in Hyper-V virtual machines, and device vendors haven’t needed to concern themselves too much about what is running inside virtual machines. Hyper-V has dealt with the indirect I/O model itself and provided drivers for those virtual devices. However, an SR-IOV device has VFs as well as PFs. With SR-IOV, a part of vendor’s hardware is exposed inside the virtual machine. As our networking code doesn’t know how to manipulate that piece of hardware directly, we need to load a vendor supplied driver in the VM.

However, the VF is not a fully-fledged device or autonomous. It cannot, for hopefully obvious security reasons, make any decisions about policy and control. It can’t instantiate itself, something else has to do that. It can’t cause another VF to be instantiated. It is transient in the overall lifetime of a given virtual machine guest operating system instance. It can only read and write parts of device configuration that the PF lets it manipulate, and it can only see the parts of networking hardware in memory space that are allocated to that VF.

While VFs are transient, the PF is always available (assuming the PCI Express device is enabled, that is). VFs cannot exist without a PF being present. As the PF runs in a trusted domain (the parent partition), the PF can be the arbiter for all policy decisions, and the control point for VF instantiation and tear-down including hardware resource allocation, working in conjunction with the rest of the Hyper-V virtualization stack.

There are also situations where the VF driver needs to communicate safely with the PF driver. These are not for the I/O path, but for policy and control logic. Microsoft considers a hardware backchannel less secure than letting Hyper-V modulate that channel of communication. By exposing interfaces in Windows for this functionality, we can validate driver behaviour by providing fuzz and penetration testing as part of driver certification, thereby encouraging vendors to take a long hard look at the security model when exposing hardware directly to a virtual machine. This is a critical tenet in our Trustworthy Computing SDL that all Windows components go through.

So what interfaces have we identified at a very high level?

To allow a PF driver to tell Windows about what SR-IOV capabilities it has
To allow a PF driver to instantiate and tear-down VFs including provisioning and de-provisioning of hardware queues and filters as required
To allow a VF driver to send policy and control logic to a PF (and back)

These are the driver interfaces which are documented in the SR-IOV NDIS functions on MSDN.

One point I want to touch on about the driver model is that we have left it up to device driver writers to decide on how they want to design their drivers. Either a split driver model where there are completely separate drivers for the PF and the VF. Or a combined driver model where there is a single driver for both the PF and the VF. Or indeed to use a virtual bus driver (VBD) if that is what they are familiar with. In other words, although there are new concepts, the step to support SR-IOV is not a huge one from a driver writer’s perspective as it builds on existing models.

When is PCI Bus not a PCI Bus?

A strange question perhaps. A traditional PCI Express device is typically enumerated on the PCI bus under a PCI Express Root Port. Here’s a screenshot from device manager using Windows Server “8” Beta in the parent partition, on a machine which has two dual-port SR-IOV capable networking devices. You can see the PF devices under a PCI Express Root Port.

And the same with a VBD driver in place from a different machine with a single dual-port SR-IOV capable networking device.

If we were to look at a virtual machine with SR-IOV devices assigned, you would see something subtly different in the device tree. This particular virtual machine has two software network adapters, each backed by a virtual function. The virtual functions are exposed on a virtual PCI Bus under VMBus as you can see in the following screen shot. Not an important point from an IT-Pro perspective, but in case you wanted to see the device hierarchy.

And again, with a VBD driver in place (with two VFs assigned to this particular VM):

This part of the blog post series started taking a look at the software side of SR-IOV, particularly from the viewpoint of device vendor driver writers. In summary, in addition to answering the “why” question, so far we’ve identified three dependencies necessary to support SR-IOV in Hyper-V in Windows Server “8” beta and hopefully explained why all of these are essential:

System hardware support in the form of an IOMMU device
A PCI Express networking device which has SR-IOV capabilities
A driver model to support both PF and VFs.

In the next part, I look at the next dependency to add to the list.

Cheers,
John.

Everything you wanted to know about SR-IOV in Hyper-V Part 2

John Howard -MSFT — Tue, 13 Mar 2012 16:19:01 GMT

In part 1, I discussed why Microsoft has been investing in SR-IOV for device I/O from virtual machines. The key points were to reduce latency, increase throughput, lower compute overhead, and for future scalability. Part two takes a look at SR-IOV from a hardware perspective.

For those who didn’t take up my offer of the light bed time reading of the SR-IOV specs (or don’t have access), let me summarize what SR-IOV is. And to be clear, when I say “SR-IOV”, take it to include closely associated specifications or additions to PCI Express specifications such as

ATS (Address Translation Services) – A PCI Express Protocol that allows a device to fetch translations from an IOMMU (Input/Output Memory Management Unit).
ARI (Alternate Routing Interpretation) – A PCI Express switch change and device change which allows a device to occupy more than eight RIDs on a single bus
ACS (Access Control Services) – A PCI Express switch feature that forces peer-to-peer traffic upstream so that it can be translated by an IOMMU.

First I’m going to say what SR-IOV isn’t. Most importantly it doesn’t refer to anything about I/O classes, and as such doesn’t have a single mention of networking, storage or other I/O classes. It doesn’t describe anything about how software should be designed to use SR-IOV capable hardware. It’s just about hardware. It doesn’t therefore either describe a driver model which is essential for a complete solution, or hardware specific nuances (one of which I mention at the end of this post) relevant to any I/O class.

The SR-IOV specs do however describe how a hardware device can expose multiple “light-weight” hardware surfaces for use by virtual machines. These are called Virtual Functions, or VFs for short. VFs are associated with a Physical Function (PF). The PF is what the parent partition uses in Hyper-V and is equivalent to the regular BDF (Bus/Device/Function) addressed PCI device you may have heard of before. The PF is responsible for arbitration relating to policy decisions (such as link speed or MAC addresses in use by VMs in the case of networking) and for I/O from the parent partition itself. Although a VF could be used by the parent partition, in Windows Server “8”, VFs are only used by virtual machines. A single PCI Express device can expose multiple PFs (such as a multi-port networking device), each (generally) independent, with their own set of VF resources. There are subtleties on multi-function devices such as ones which support, for example, iSCSI, Ethernet and FCoE, but this is beyond the depth of this series of posts and the approach differs between hardware vendors.

It is important to note that VFs are hardware resources. Because these are hardware resources, there are constraints on the number of VFs which are available on any particular device. The actual number will differ across vendors and devices. You can expect that as hardware moves forward through newer revisions, the trend will be to offer more VFs per PF. Typically we are seeing devices offering 16, 32 or 64 VFs per PF in 1st generation 10 GigE SR-IOV enabled networking hardware.

VFs alone aren’t sufficient to be able to securely allow a VM direct access to hardware. Traditional PCI Express devices generally “talk” in system physical address space (SPA) terms. As you may be aware, we don’t run guest operating systems in SPA, we run them in guest physical address space, or GPA. So there has to be something which translates (and ideally caches) addresses for DMA transfers. This is DMA remapping. In addition, for security reasons, we require hardware assisted interrupt remapping. For those who are want to learn more about the hardware side of this, see this page about VT-d for Intel, or this page for AMD-V for AMD. There are plenty of specs to read for the inquisitive reader!

From this point on in this series, I’m going to generically use the term IOMMU to refer to hardware capabilities which provide interrupt and DMA remapping.

To be clear, as I haven’t said it explicitly otherwise, an SR-IOV device with a suitable driver can be used as a regular I/O device outside of virtualisation. It probably wouldn’t take advantage of the additional hardware capabilities without virtualisation present, but it still can be used as a regular I/O device. Further, the device does not require the presence of “IOMMU” hardware to be used in this manner.

Although I’ve referred to networking a few times so far, I’ve also said SR-IOV from a spec standpoint doesn’t mention anything about an I/O class. When we (the Hyper-V team) looked at where the biggest gains were in using SR-IOV, it was clear to us that the overhead of storage I/O was significantly less than that of networking I/O. Hence for Windows Server “8” we have exclusively worked on SR-IOV for networking as the only supported device class.

Although it may not be immediately obvious, for a NIC vendor to create an SR-IOV capable PCI Express device, it’s not sufficient to follow the SR-IOV specifications alone. One reason is that the NIC has to do networking to and from multiple sources (PF and VFs), as well as on the wire. To enable Ethernet frames to be routed between two VFs, for example, most parts of an Ethernet switch have to be embedded onto the physical NIC. None of this is present in the SR-IOV specifications.

So now we’ve covered SR-IOV from the “why” perspective in part 1, and the hardware perspective in this part. Part 3 will take a look at the software perspective of supporting SR-IOV in Windows Server “8”.

Cheers
John.

Everything you wanted to know about SR-IOV in Hyper-V. Part 1

John Howard -MSFT — Mon, 12 Mar 2012 19:07:00 GMT

Now that the veil of silence has been lifted with the release of Windows Server “8” beta , I’m able to tell you a little more about what I’ve been working on for 5 years now. An inside joke is that it’s a feature with two checkboxes in the UI of Hyper-V Manager, so about 2½ years per checkbox! Well clearly it’s a little more than that! A lot more, actually!!

SR-IOV stands for Single-Root Input/Output (I/O) Virtualization. It’s is a standard defined by the PCI Special Interest Group. If you work for one of the member companies who have access, and are after some light bedtime reading, the specs are available on their website.

To keep it at a high level (as in all you probably need to know to use the feature), SR-IOV is a standard which describes how a PCI Express device can be constructed so that it works well with modern virtualization solutions. You may be wondering, what does “work well” mean when we already have a great device I/O sharing model present in Hyper-V in both Windows Server 2008 and Windows Server 2008 R2? Good question. Before I answer that, let’s take a diversionary look at a diagram many of you will have seen variations of dozens of times before.

In the above diagram (I’m using networking for my example, but the same principles apply to storage), the physical device is “owned” by the parent partition. The parent is the arbiter for all traffic originating from VMs to the outside world and vice versa. The parent is also responsible for all policy decisions regarding how the device behaves such as link speed in the case of a networking device.

Emulated versus Software Devices
Virtual machines “see” either emulated devices (such as the Intel/DEC 21140), or software based devices (commonly referred to as either “synthetic” devices, a term I personally try and avoid ever using, or “paravirtualised” devices) which are designed to work well in a virtualised environment. In both these cases, these devices aren’t “real” devices physically present in the actual hardware you can touch. In fact, in the case of a software based device, it’s a completely made up fabricated device. You can’t go to your local store and buy one as it doesn’t exist in the physical world. Software based devices take advantage of our high-speed inter partition communication mechanism, VMBus, to efficiently pass data between the parent partition and a virtual machine. Software based devices are far more efficient than emulated devices for four main reasons:

First, we don’t (generally) need the Hypervisor to be involved in the hot-path for transfer of data. An emulated device requires many Hypervisor intercepts for every single I/O making it very expensive from a performance perspective which is why all of our supported operating systems also include drivers for software devices.
Second, VMBus uses shared memory buffers and memory descriptors to move data between the parent partition and the virtual machine. Shared memory access across partitions is extremely fast, especially as system architecture and hardware technologies have improved over recent years.
Third, we don’t need to literally emulate a physical device in user mode in the worker process, instruction by instruction, as we do for emulated devices. Software devices do not require emulation.
Fourth, VMBus interfaces (for at least networking and storage) runs in kernel mode in both the VM and the parent partition. We don’t need to transition up to user mode in the parent partition to complete I/Os, consuming additional compute cycles.

While software based devices work extremely efficiently, they still have an unavoidable overhead to the I/O path. For example, for security reasons, we sometimes (but not always, depending on direction and I/O class) need to copy data buffers between a virtual machine and the parent partition. No software can run with zero overhead no matter how much fine tuning is applied. And believe me, we spend a lot of time tuning Hyper-V for performance. Consequently, software based devices introduce latency, increase overall path length and consume compute cycles.

Ultimately, the day will come where software alone will not be able to keep up with link speeds. With RDMA, we are already getting close. One very approximate figure, and deliberately so, is looking at how much compute resource is required for Ethernet based network I/O. This depends hugely on processor class and vendor driver, but today a single core could be consumed by between 5 and 7 GB/sec of networking traffic generated by virtual machines using Windows Server 2008 R2 SP1. Furthermore, as line rates increase, 40 Gigabit Ethernet and 100 Gigabit Ethernet already standardised, we have to look at how we can scale Hyper-V I/O effectively in a virtualised datacentre.

Now I’m not saying that SR-IOV is only useful in 40 and 100 Gigabit environments. Absolutely not! But with 10 GigE hardware rapidly being adopted, the time is right to look at alternate more efficient mechanisms for device I/O which will continue to scale well in the future.

To answer therefore what “works well” means as I mentioned earlier, it means a secure device model which has, relative to software based device sharing I/O, lower latency, higher throughput, lower compute overhead, and scales well in the future. These are all met by SR-IOV.

So now you understand a little more about why Microsoft has been investing in SR-IOV and Hyper-V in Windows Server “8”, in the next part, I’ll start getting into the detail.

Cheers
John.

Hyper-V Manager for Windows 7 Service Pack 1

John Howard -MSFT — Fri, 08 Apr 2011 03:30:17 GMT

The Remote Server Administration Tools (RSAT) for Windows 7 SP1 are now available for download. These include the latest version of Hyper-V Manager which supports both Dynamic Memory (DM) and RemoteFX.

Here's the link: http://www.microsoft.com/downloads/en/details.aspx?FamilyID=7d2f6ad7-656b-4313-a005-4e344e43997d

Cheers,
John.

Hyper-V BPA Update

John Howard -MSFT — Fri, 25 Feb 2011 00:52:12 GMT

Today we released an update to the Hyper-V BPA (Best Practice Analyser). This fixes some issues in the previous BPA and also correctly handles the new Dynamic Memory and RemoteFX features in SP1. No new rules have been added. To install the update, you must already have the Hyper-V BPA installed (http://support.microsoft.com/kb/977238). Note that both the original BPA and this update apply to Windows Server 2008 R2 and Microsoft Hyper-V Server 2008 R2.

While I recommend you have SP1 installed, this update also applies to R2 RTM Hyper-V. A quick summary of the fixes:

Microsoft Hyper-V Server is no longer flagged as not being a Server Core installation
Some ECC RAM machines were reported as non-ECC as some non-ECC firmware RAM was being included in the check
We have not included the File Service role feature in the scan. The reason is that we have observed certain products (Data Protection Manager as an example) legitimately creating shares for adminstrative purposes. The BPA no longer flags the presence of a file share as another role being installed. However, please do NOT use your Hyper-V machine as a general purpose file server!
If you have SP1 and enable RemoteFX, we won't flag the machine as uncompliant due to another role being installed.
In the case of Windows 7 VMs only, we have increased the permitted consolidation ratio to 12:1 and updated the BPA accordingly to check for compliance.
As SP1 changes the minimum supported memory for Windows 7 VMs if DM is enabled, the BPA has been updated accordingly.

http://support.microsoft.com/kb/2485986 is the place for all the information on the changes and the download link.

Cheers,
John.

Explaining the Hyper-V authorization model, part six

John Howard -MSFT — Mon, 14 Feb 2011 23:00:00 GMT

Hyper-V uses a role based authorisation model for access checks. This series of articles takes a look at the model; defines the available primitives; and walks through a couple of examples. (Oops, a post I meant to publish back in October last year, only just realized I didn't make it live!)

Quick links: Part1; Part 2; Part 3; Part 4; Part 5;

This part examines a solution I use at home to make sure that critical production VMs can’t be ‘snapshotted’. This makes use of the new operation (355) in the R2 release. The approach is simply to group VMs into either “live” or “test” buckets. All operations are permitted for VMs which are in the test bucket. A subset of operations is permitted for VMs which are in the live bucket.

Here’s a screenshot of Hyper-V Manager running one of my servers: I’ve also chosen to name the VMs with a “Live:” or “Test:” prefix.

The first thing I need to do is manipulate the AZMan store to create a new scope called “Test”, and within that, create a role Definition “Administrator (Test VMs)” which is authorized to all operations.

Within that scope, I create a new role assignment “Administrator (Test)”, and link it to the “Administrator (Test VMs)” role definition in the “Test” scope

I then add user accounts as needed to this role assignment that should have full access to test VMs.

Next, back to the default scope, and alter the “Administrator” role definition to remove the “Allow Virtual Machine Snapshot” operation (hit ‘Remove’ at the following dialog). In effect, the default scope is going to become the scope where live VMs reside which can’t be snapshotted.

Almost there – at this stage, all VMs are in the default scope. Let’s verify that snapshots can’t be taken by trying to snapshot one of the VMs I have. I picked a “test” one as that’s one I want to be able to snapshot in the next step.

The last part of the configuration is to move each of the test VMs into the “Test” scope. I’ll use the same SetScope.vbs script from part three of this series. Again, look for the “0” on the last line of output for success.

Let’s try that snapshot operation again on the “Test: R2 Core” VM again:

As you can see, that succeeds. So once all test VMs are moved to the “Test” scope, I’ve achieved the goal: Snapshots cannot be taken of live VMs, but snapshots can be taken of test VMs.

Cheers,
John.

Updated Linux Integration Services are available

John Howard -MSFT — Thu, 29 Jul 2010 21:53:00 GMT

A new version (2.1) of the Linux Integration Services for Hyper-V have just been made available. This new version has some significant improvements. From the driver side, new optimized storage and networking drivers have been introduced (aka “Synthetic” drivers which utilize VMBus, a key component in the Hyper-V architecture, essential for efficient and fast device I/O without the need for emulation). Supporting VMBus allows us to boot Linux guests faster. Integration components have been added in the form of time synchronization, heartbeat monitoring and shutdown capability.

And the big one that everyone I talk to has been asking for…. up to 4-way SMP Linux guests.

For more information and details about the supported guests, check out the blog post on my team blog site. You can download the drivers from here.

Cheers,
John.

Avoiding UAC prompt starting Hyper-V Manager

John Howard -MSFT — Wed, 07 Jul 2010 23:13:01 GMT

An interesting question was posed to me a couple of days back about why when starting Hyper-V Manager, you get a UAC prompt, yet you don't with most other Microsoft Administrative MMCs. You should note that the workaround in here only applies to Windows 7/Windows Server 2008 R2, and is not a supported workaround.

This post has more information about what changed in UAC in Windows 7 & R2. Unfortunately, the simple fact of time and the team learning of the change too late meant that we shipped Hyper-V the same as in Vista/2008 by placing virtmgmt.msc, in \program files\Hyper-V. As that location is not, in Marks terminology, a "secure location", or at least not one known about by MMC, you get the elevation prompt when starting Hyper-V Manager.

The workaround is a simple one.

First, from an elevated command prompt, copy \program files\Hyper-V\virtmgmt.msc to \windows\system32.

Next, open the properties for the shortcut to Hyper-V Manager located under Administrative Tools. The target will be %windir%\system32\mmc.exe "%ProgramFiles%\Hyper-V\virtmgmt.msc". Update this to %windir%\system32\mmc.exe %windir%\system32\virtmgmt.msc. Do not change the "Start in" location though.

From

Note that double quotes are not necessary. When you hit enter, you will get a UAC prompt to confirm the update.

And that's it - starting Hyper-V Manager should no longer prompt you for elevation.
Cheers,
John.

Getting event log contents by email on an event log trigger

John Howard -MSFT — Thu, 17 Jun 2010 00:22:20 GMT

This one was actually pretty simple to work out, but it did have me flummoxed to start with. Here’s the scenario, I wanted to get an email when an event log entry was triggered. But, I also wanted the contents of the event log entry. I’ve been meaning to document this for ages, but never seem to find the time!

So here’s an example of the in-box functionality vs. a simple bit of bolt-on customization. In this example, I’ll use Event 20274 for RemoteAccess on a Windows Server 2008 R2 box running TMG 2010. This particular event is logged when an inbound VPN connection is established, and the body of the message says who connected, on what port, and what IP address they have been allocated.

First, inbox functionality. Establish the VPN, and find the event in the event log.

Down in the bottom right, choose “Attach Task To This Event….”, and walk through the wizard. On the first screen, give it an appropriate name such as “A user connected through VPN”. On the action page, select send an email. On the Send an email page, fill in the appropriate information for From/To/Subject/Text and SMTP Server. What you’ll notice is that there’s nowhere to specify what goes in the body. But you can include a static attachment, but that doesn’t serve our needs

Finish the wizard, and connect again through VPN to see what email comes through. Not particularly useful. Not yet, anyway.

Now if you go into task scheduler, and drill down through Task Scheduler Library then to Event Viewer Tasks, you’ll see a new item. If you go into the properties of the task, you’ll see there’s no way to include the text of the event log in the message.

So step back a second, and ask “what’s the easiest way to get the last instance of event 20274 firing in the System event log?”. The answer (or an answer) is wevtutil. Here’s a command that will do that (note all on one line):

wevtutil qe System "/q:*[System [(EventID=20274)]]" /f:text /rd:true /c:1

Running that in a command prompt will yield the following:

Perfect, so that’s what I want emailed to me. So let’s create a quick batch file which will get the above information and put it in a file. I just called it query.cmd and saved it on my desktop for convenience (again, the wevtutil command is all on one line).

del %temp%\query.txt
wevtutil qe System "/q:*[System [(EventID=20274)]]" /f:text /rd:true /c:1 > %temp%\query.txt

With that done, let’s revisit the properties of the task and look at the Actions tab. Let’s add an item to run this batch file, and put it top of the list.

Now we need to look at the properties of the “Send an e-mail” option. Remember there was an “Attachment” setting. Well conveniently, we have a file which contains the information we need, %temp%\query.txt now. Simply put “C:\Users\tmgadmin\AppData\Local\Temp\query.txt” in that box. (Obviously replace the username/location as appropriate). I’m also going to remove the body of the message.

So what does the email look like now if I establish a VPN?

Exactly what I wanted! Hope that helps someone.

(And before you ask, the only link this post has to Hyper-V is that my TMG and Email servers are Hyper-V VMs).

Cheers,
John.

PS – yes, I realize this may not be perfect if two users connect at exactly the same time, or in your use case that multiple events fire at the same time, but I’ll leave that as an exercise for the reader to solve :)

Announcing NVSPBind

John Howard -MSFT — Mon, 25 Jan 2010 23:33:00 GMT

A quick post to announce the availability of a new utility written by a colleague of mine in the Hyper-V team, Keith Mange. NVSPBind (Network Virtual Service Provider Bind) overcomes a shortfall that many people hit in server core installations of Windows Server 2008, Windows Server 2008 R2 and Microsoft Hyper-V Server.

In a full installation of Windows, it is possible to enable or disable protocols from a network adapter using the network connections applet in the control panel (aka ncpl.cpl) by simply checking or unchecking protocols as necessary.

This applet isn't available in server core and there is no in-box utility to perform this action. Let's walk through a really simple example and say we want to disable File and Printer Sharing for Microsoft Networks on an adapter.

The first step is to obtain the GUID which uniquely identifies the adapter. This can be achieved by running nvspbind with no command line parameters. Here's the truncated output from a test machine showing the NIC I'm looking for:

{F93672D9-9085-4EEF-9669154AD4391ED7}
"pci\ven_8086&dev_10c9&subsys_a03c8086"
"Intel(R) Gigabit ET Dual Port Server Adapter":
   enabled: ms_netbios       (NetBIOS Interface)
   enabled: ms_server        (File and Printer Sharing for Microsoft Networks)
   enabled: ms_pacer         (QoS Packet Scheduler)
   disabled: ms_ndiscap       (NDIS Capture LightWeight Filter)
   enabled: ms_wfplwf        (WFP Lightweight Filter)
   enabled: ms_msclient      (Client for Microsoft Networks)
   enabled: ms_tcpip6        (Internet Protocol Version 6 (TCP/IPv6))
   enabled: ms_netbt         (WINS Client(TCP/IP) Protocol)
   enabled: ms_smb           (Microsoft NetbiosSmb)
   enabled: ms_tcpip         (Internet Protocol Version 4 (TCP/IPv4))
   enabled: ms_lltdio        (Link-Layer Topology Discovery Mapper I/O Driver)
   enabled: ms_rspndr        (Link-Layer Topology Discovery Responder)
   enabled: ms_pppoe         (Point to Point Protocol Over Ethernet)
   enabled: ms_ndisuio       (NDIS Usermode I/O Protocol)
   disabled: vms_pp           (Microsoft Virtual Network Switch Protocol)

The GUID is right at the top of the output (starting "{F9367"). The protocol I want to unbind in this example is ms_server (the short name for File and Printer Sharing for Microsoft Networks).

This is achieved using the -d parameter as follows:

C:\>nvspbind -d {F93672D9-9085-4EEF-9669154AD4391ED7} ms_server
Hyper-V Network VSP Bind Application 6.1.7672.0.
Copyright (c) Microsoft Corporation. All rights reserved.
acquiring write lock...success

Adapters:
{F93672D9-9085-4EEF-9669154AD4391ED7}
"pci\ven_8086&dev_10c9&subsys_a03c8086"
"Intel(R) Gigabit ET Dual Port Server Adapter":
    unbinding ms_server from Intel(R) Gigabit ET Dual Port Server Adapter
    unbinding ms_server from Intel(R) Gigabit ET Dual Port Server Adapter
    unbinding ms_server from Intel(R) Gigabit ET Dual Port Server Adapter
    unbinding ms_server from Intel(R) Gigabit ET Dual Port Server Adapter
    unbinding ms_server from Intel(R) Gigabit ET Dual Port Server Adapter
    unbinding ms_server from Intel(R) Gigabit ET Dual Port Server Adapter
applying changes...
cleaning up...releasing write lock...success
finished
C:\>

NVSPBind also has the ability to enable bindings, bind or unbind the Hyper-V network switch protocol from a NIC and repair bindings on a NIC. More information on scenarios and usage is in the package which can be downloaded from http://code.msdn.microsoft.com/NVSPBind.

As with all utilities which change network configurations, be extremely careful as you may disrupt or even lose network connectivity if you are managing a machine remotely. It may be handy to have Keith's other utility around just in case. http://code.msdn.microsoft.com/nvspscrub

Cheers,
John.

Explaining the Hyper-V authorization model, part five

John Howard -MSFT — Fri, 09 Oct 2009 21:51:00 GMT

Hyper-V uses a role based authorisation model for access checks. This series of articles takes a look at the model; defines the available primitives; and walks through a couple of examples. (I actually wrote most of this many months ago – only finally found the time to post it up!).

Quick links: Part1; Part 2; Part 3; Part 4

This post describes a change to the authorisation model in Hyper-V for Windows Server 2008 R2. If you recall from part one, I mentioned that there are 33 operations defined in AZMan for Windows Server 2008, and 34 operations for Windows Server 2008 R2.

The new operation has ID 355, ‘Allow Virtual Machine Snapshot’.

Why (to me) is this useful? Have you ever been confronted with a screen such as the following where you want to make Hyper-V Manager the foreground application, but accidentally hit the ‘Snapshot’ action in the MMC? I assure you, I have, several times.

The problem with accidentally hitting that action is that you could now find your production virtual machine using a differencing disk, with reduced performance (at least in v1 – not the case in R2), or the possibility of physical disk space running out. Further, to merge the changes back to the parent VHD so that a differencing disk is no longer being used, you need to delete the snapshot, shut down the virtual machine, wait for the merge to complete and then restart the VM. This is particularly painful when the VM in question is your ISA server for outbound Internet connectivity, or your Exchange server your clients (wife and children in my case) are using for email?

In part six, I’ll look at a solution that I personally use at home on my Windows Server 2008 R2 production environment that builds on what’s been learnt so far to ensure that I can’t accidentally snapshot critical production VMs, but am able to snapshot test VMs to my hearts delight.

Cheers,
John.

Explaining the Hyper-V authorization model, part four

John Howard -MSFT — Fri, 18 Sep 2009 22:29:00 GMT

Hyper-V uses a role based authorisation model for access checks. This series of articles takes a look at the model; defines the available primitives; and walks through a couple of examples. (I actually wrote most of this series many months ago – only finally found the time to post it up!).

Quick links: Part1; Part 2; Part 3; Part 4

In parts two and three, I walked through a specific scenario. However, you’re probably asking after having read them how I knew what operations are needed, and when, and in what scope. Well, luckily I can walk across the corridor and speak to our development team. Obviously this isn’t practical for most of you reading this, but there are, of course other ways of discovering which access checks are failing. A great resource recently published on Technet is http://technet.microsoft.com/en-us/library/dd282980(WS.10).aspx. However, there’s a sneakier way…. Let’s take a step back through part three, and delete the role assignment and role definition “Service Access” I created to cause a deliberate access check failure.

Next, I turned on auditing for object access. (I’m ignoring the fact that local policy may be over-ridden by group policy in a domain environment – this walkthrough so far is entirely on a workgroup configuration). Start the Local Security Policy snap-in under Administrative tools, and navigate to Security Settings/Local Policies/Audit Policy and change to auditing Success and Failure.

Once done, log back on as user Joe or John and start Hyper-V Manager to validate the user gets the familiar ‘You do not have the required permission to complete this task.’ message.

Now log on as a user with local admin rights and start the event viewer. Select the Security log under Windows Logs, and optionally apply a filter for just events 4665-4667 (actually just 4666 is probably enough).

What you’ll see is the following Audit Failure message for event ID 4666: Joe failed an access check to operation Read Service Configuration (operation ID 100) in scope “blank” (ie the default scope).

So with that knowledge, it’s easy to debug issues in authorisation models that you develop.

In the next part of this series, I’ll look through a really useful change in Windows Server 2008 R2 (and Microsoft Hyper-V Server 2008 R2) which leads me in to a walkthrough in part six of an authorisation policy example I use on my home servers.

Cheers,
John.

Explaining the Hyper-V authorization model, part three

John Howard -MSFT — Thu, 10 Sep 2009 04:52:00 GMT

Hyper-V uses a role based authorisation model for access checks. This series of articles takes a look at the model; defines the available primitives; and walks through a couple of examples. (I actually wrote most of this series many months ago – only finally found the time to post it up!).

Quick links: Part1; Part 2; Part 3; Part 4

In part two, I started creating the scenario of separating the view two users have when opening Hyper-V Manager so that they only see their own VMs. To do that, I created two VM Scopes, one for each user, and moved the users’ VMs to the required VM scopes. I mentioned that some more steps were still required. This part of the series walks through those steps.

The first part of the additional steps I’ve pretty well covered to death in my remote management of Hyper-V series. As my two users, John and Joe are not local administrators; they need to be granted explicit access to WMI namespaces (and Distributed COM Users if managing remotely). By far the easiest way to achieve this is using HVRemote. (Note, I’m assuming you’re following best practice, and using remote management as the server is running Hyper-V Server, the standalone SKU, or a Server Core installation of Windows Server 2008/2008 R2.)

From an elevated command prompt when logged on as a local administrator, run

cscript hvremote.wsf /add:john

(Use /add:domain\account if the Hyper-V machine is domain joined)

If you look at the output carefully, note the line that says “Adding john to AZMan role Administrator” near the bottom. Apart from a typo I need to correct in a future version, what HVRemote has done is add john to the ‘Administrator’ role assignment in the default scope. This is simply a limitation in HVRemote. At the time of writing, HVRemote cannot cope with VM Scopes. (In fact it is hard coded to always update the role assignment called ‘Administrator’ in the default scope – on my big list and will be covered in the future).

If John and Joe are now administrators in the default scope, we’ve not performed any separation as administrators in the default scope can view all VMs. There are two ways to resolve this. Either we update policy using Authorisation Manager to undo the generalisation HVRemote has made here, or use a parameter available in HVRemote when adding the account.

Method 1 Use Authorisation Manager

Select the Role Assignment ‘Administrator’ in the root scope to find the user which has been added.

Right click on the user and choose delete (or simply hit the delete key)

Method 2 Use a parameter to hvremote

*Caveat this may not work in future releases – it does as of version 0.7 though.

From an elevated command prompt when logged on as a local administrator, run

cscript hvremote.wsf /add:john /noazman

(Use /add:domain\account if the Hyper-V machine is domain joined)

If you compare this output to the previous hvremote output, notice there is no role assignment update in AZMan.

The last part to get our user separation in place require a little thought to get your head around, and a little knowledge of the Hyper-V design.

We have a service called VMMS (Virtual Machine Management Service). There are two operations in our authorisation model which are required to be able to perform operations on the VMMS. VMMS always performs its access checks for these operations in the default scope.

The operations are

Read Service Configuration, and
Reconfigure Service

What this means is that the users I’m separating, John and Joe, must be authorized to these operations in the default scope. It is not sufficient to just have them an ‘administrator’ in the VM Scope. Based on our knowledge from the previous parts, this is easily achieved:

Create a Role Definition ‘Service Access’ containing the two operations in the default scope
Create a Role Assignment ‘Service Access’ linked to the ‘Service Access’ role definition in the default scope
Add John and Joe to the role assignment ‘Service Access’ in the default scope

Authorisation manager should look like the following when done:

All that remains validate the configuration is to log on as those users and start Hyper-V Manager (or use Hyper-V Manager remotely in the case of a server core or Microsoft Hyper-V Server installation).

Here, I’m logged on as John

And here, I’m logged on as Joe

So that works as expected. With the information so far, I hope I’ve provided everything you need to enable you to build a model which makes sense for your own unique implementation. It’s a question of sitting down and working through how to map your organisational needs into authorisation model primitives. There is no single right answer which fits everyone, so building a mapping is not something I can help you with!

In the next part of this series, I’ll take a look at how you could debug issues with a custom authorisation model you develop.

Cheers,
John.

Explaining the Hyper-V authorization model, part two

John Howard -MSFT — Wed, 02 Sep 2009 21:22:00 GMT

Hyper-V uses a role based authorisation model for access checks. This series of articles takes a look at the model; defines the available primitives; and walks through a couple of examples. (I actually wrote most of this series many months ago – only finally found the time to post it up!).

Quick links: Part1; Part 2; Part 3; Part 4

Part one provided information on the primitives available in the AZMan model and looked at the out-of-box Hyper-V configuration. Building on that information, part two takes a deeper look at scopes.

In part one, I talked about the top level scope (aka root or default scope) as the place where global policy is defined. I mentioned that you can also define more constrained scopes and place virtual machines in those scopes.

The first question to answer is “How can you create a ‘Virtual Machine’ scope?”. Scopes exist at an application level. You can either right-click on an application to create a new scope, as shown in the screenshot below, or use a script if you prefer automation (as I do). (If you’re interested in the specifics of API calls, take a look at http://msdn.microsoft.com/en-us/library/aa375769(VS.85).aspx).

Note that the script is the barest minimum – obviously I would recommend you make something more resilient for general use.
Save the following code as “CreateScope.vbs”.

' Make sure the script is passed a scope to create

szScope = wscript.arguments.named("scope")

if szScope = "" then

    wscript.echo "CreateScope /scope:<name>"

    wscript.quit

end if

' Need to have an object referencing the store

set oAuthStore = _

   CreateObject("AZRoles.AZAuthorizationStore")

' Initialise the store so that we can update it

oAuthStore.Initialize 0, _

   "msxml://C:\ProgramData\Microsoft\Windows\" & _

   "Hyper-V\InitialStore.xml"

' Open the Hyper-V services application in the store

Set oApplication = _

    oAuthStore.OpenApplication("Hyper-V services")

' Create a new scope

Set oNewScope = _

    oApplication.CreateScope2(szScope)

' Submit it to the store

oNewScope.Submit

To create a scope called “My test VM scope”, from an elevated command prompt, type

cscript createscope.vbs /scope:“My test VM scope”

If you already have the Authorisation Manager MMC open after walking through part one, you need to reload the authorisation store by right-clicking on InitialStore.xml in the treeview on the left and selecting Reload. If the Authorisation Manager MMC is not open, open it now and load InitialStore.xml.

When you expand out the tree, you’ll see that a new scope called “My test VM scope” has been created:

You can also see that a “VM Scope” has the same primitives available under it as the “Default” Scope – Groups, Role Definitions, Task Definitions and Role Assignments. You can use the MMC to create role definitions; assign operations to role definitions; create role assignments; link role assignments to role definitions; and assign accounts to role assignments at both the default scope level and at the VM Scope level.

Side note:

Personally, to avoid confusion, I would avoid using role definitions in a VM scope unless you really need to keep a role definition so specific that is has to be tied to a particular VM scope. There is little reason to not create all the role definitions at the default scope level.

At this point, you probably have a question: “Why would I need a ‘Virtual Machine’ scope?” And a great question it is, too. To answer it, let’s consider the following simple scenario:

You have a shared Hyper-V machine. It is used by two users called “John” and “Joe”. Both John and Joe have a single VM of theirs on that server called “Johns VM” and “Joes VM” respectively.

You want the system configured so that John cannot even see that Joe is a user or has VMs on that machine, and visa-versa. John must be able to perform all operations on his virtual machine, and Joe must be able to perform all operations on his virtual machine. Neither Joe or John should be administrators on the physical machine.

Without virtual machine scopes, this is not possible. Let’s work through how you would configure that scenario. I’m going to start with the blank InitialStore.xml again for this. Using CreateScope.vbs, create scopes called “Johns VM Scope” and “Joes VM Scope”. Reload InitialStore.xml in Authorisation Manager.

Within each of these new VM scopes, I’ll now use Authorisation Manager to create a new role assignment called ‘Administrator’ (not to be confused with the default scope role assignment ‘Administrator’ – AZMan permits role assignments with the same name in different scopes), and link it to the default scope ‘Administrator’ role definition.

Right-click “Role Assignments” under the VM scope and choose New Role Assignment…

In the dialog, select the role definition ‘Administrator’ in the default scope(called ‘Where Defined: Application’ in the UI) by checking it, then hit OK.

The tree view should look something like this:

The next step is to put “John” into the Administrator role assignment in the VM Scope ‘John’s VMs” and to put “Joe” into the Administrators role assignment in the VM Scope ‘Joe’s VMs”. To do this, right-click the newly added role assignment and choose Assign Users and Groups, then From Windows and Active Directory.

After making the necessary changes, your policy store should look like the following:

Important to remember and note is that neither John or Joe are local administrators on the machine. Let’s take a step forward and assume that John and Joe have already created a virtual machine. In the screenshot below, I’m logged on as the local administrator. Remember that built in administrators are administrators in the default scope in the default policy store, and hence can see both virtual machines.

The magic sauce needed to make the separation is to move Joes VM into the scope ‘Joes VM Scope’ and move Johns VM into the scope ‘Johns VM Scope’. Now as much as I write my own scripts for just about everything, there are certain times where it makes no sense to reinvent the wheel. If you download BackupVMsAndScopeScripts.zip and expand the files, there is a script called SetScope.vbs

Run the script:

cscript setscope.vbs “Johns VM” “Johns VM Scope”
and cscript setscope.vbs “Joes VM” “Joes VM Scope”

You’ll get a big spew of output from each command. I’ll leave it an exercise for the reader to modify the script to their needs or develop their own. You want to look at the very last bit of the output which will say “0” if the update succeeded.

That’s it from a VM configuration standpoint, but there are still some nuances which needs resolving before John and Joe can use Hyper-V Manager to get their custom view of only their VMs. As it’s involves several steps, I’ll cover this in part three.

Cheers,
John.

Explaining the Hyper-V authorization model, part one

John Howard -MSFT — Tue, 01 Sep 2009 05:28:00 GMT

Hyper-V uses a role based authorisation model for access checks. This series of articles takes a look at the model; defines the available primitives; and walks through a couple of examples. (I actually wrote most of this series many months ago – only finally found the time to post it up!).

Quick links: Part1; Part 2; Part 3; Part 4

As the term ‘Role Based Authorisation Model’ implies, Hyper-V has an authorisation layer which performs access checks to grant or deny an account access to operations based on roles the account is a member of. That is not to say everything in Hyper-V has an authorisation protection layer – we also use traditional NT ACL-based security mechanisms. However, this series of articles concentrates just on the authorisation model.

A term I should introduce at this point is “AZMan”. AZMan is, in short, an engine and toolset for making role based access checks and defining policy. AZMan is a component built into Windows. Hyper-V uses AZMan to control role based authorisation.

When you install Hyper-V, the system is configured with a policy store, the policy store being nothing more than a file on disk called ‘InitialStore.xml’. InitialStore.xml contains the most simple of authorisation policies: local administrators are authorised to perform all operations protected by a policy check.

There are two registry keys Hyper-V uses to define attributes about the policy store. They are both under HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Virtualization.

StoreLocation points to a file called InitialStore in a hidden directory c:\ProgramData.
ServiceApplication defines which application in the policy store is used.

Before looking at the contents of InitialStore.xml, let’s define some primitives which Hyper-V uses in AZMan.

(Note that AZMan is a very flexible model, and Hyper-V does not use all the primitives available in AZMan. Also these articles do not cover more advanced modeling capabilities where you can, to some extent, build nested models).

Application

Policy stores define the authorisation model for zero or more applications. An application is top-level container which contains all the other primitives used by an application. Examples of applications could be ‘My Financials Application’ or (not by any coincidence!) ‘Hyper-V services’.

Operations

Operations are specific items or actions being guarded by an access check in an application. For example, when a user tells a system to “Create a Virtual Machine”, Hyper-V makes an access check.

Role Definitions

As the name implies, this is the definition of a specific role, such as a Hyper-V administrator, or a Hyper-V Network Administrator, or standard user. It is defined by a name and zero or more operations which are permissible in this role definition. So with the examples I just gave, you could choose to setup the store such that an administrator role definition contains all operations, a network administrator role definition contain the operations for altering, creating or deleting virtual networks, and standard users can only connect and interact with virtual machines.

Role Assignments

Role assignments are where users and groups are placed or “assigned” in the model. A role assignment can be stand alone or linked to one or more role definitions.
A standalone role assignment doesn’t make too much sense, as without any role definition links, it is nothing more than an orphaned grouping of accounts.
When you link a role assignment to a role definition, you are saying that the accounts in the role assignment have would pass access checks for the operations defined in the linked role definition. If you link a role assignment to multiple role definitions, accounts in the role assignment would pass access checks for the superset of operations defined by all the linked role definitions.
Role assignments can be created at multiple levels or “scopes”.

Scopes

Scopes are a more advanced feature in the authorisation model. Scopes can be thought of as the “level” where role definitions, role assignments and other AZMan primitives reside. Each AZMan policy store contains a single top level scope. A good way to think of the top level scope is the place where global policy is defined.

Side note

Internally on the Hyper-V team, this tends to be referred to as the ‘default’ or ‘root’ scope. Whether that is correct or not in AZMan terminology is another question! I believe the correct term is “Application” scope.

You can also create additional “more constrained” scopes, and place virtual machines in them. For the purposes of Hyper-V, it makes sense to think of these as VM specific scopes, or VM Scopes for short. Using VM scopes is a topic I’ll cover in more detail in later parts of this series.

So with knowledge of those primitives, let’s take a look at the out-of-box policy store. To use the management tools for AZMan requires a full installation of Windows Server rather than server core. (Or you can take a copy of the file across to a separate Vista or Windows 7 installation). As the policy store is in a hidden and ACL’d directory, you need to be a local administrator to open the file.

Start azman.msc

Right click on Authorisation Manager in the tree view on the left and select Open Authorisation Store. Navigate, or enter the path, to InitialStore.xml (again, note \ProgramData is a hidden directory – you can type it in though).

Once open, the first thing to notice is that there is a single application defined: ‘Hyper-V services’.

This matches the registry key ‘ServiceApplication’ in the screenshot higher up this article. Next, I’ve expanded out a couple of nodes so that you can see where Role Definitions and Role Assignments fit in to the hierarchy.

(Note the Hyper-V authorisation model does not use Task Definitions or Authorisation Rules, and I won’t be talking about Groups in this series of articles)

You can see that we have a single role definition ‘Administrator’ and a single role assignment ‘Administrator’ defined. First, examine the ‘Administrator’ role definition by right clicking on it, selecting properties and choosing the Definition tab.

This is where we can see which operations are covered by a role definition. There are 33 operations in Hyper-V in Windows Server 2008 and Microsoft Hyper-V Server, and 34 operations in Windows Server 2008 R2 and Microsoft Hyper-V Server R2.

If you select a role assignment, Authorisation Manager displays a list of accounts who are members of that role assignment – the default being the builtin administrators group.

As an aside – if you are walking through this and are using a domain joined machine (the above screenshot is from a workgroup machine), the default accounts listed in the “Administrator” Role Assignment will be domainname\administrators. This is actually a bug (as far as I can tell) in the AZMan console. You can verify this by opening InitialStore.xml in an editor:

You’ll notice that the SID listed is S-1-5-32-544 which is the SID (Security Identifier) of the builtin\administrators group - http://support.microsoft.com/kb/163846 has more information.

So that explains how the default policy in Hyper-V is setup such that local administrators have access to all operations. With an understanding of the primitives, it is trivial to extrapolate how to modify the model to grant other accounts full access to Hyper-V without needing to be local administrators on the Hyper-V machine itself – you simply need to add accounts to the ‘Administrator’ role assignment in the default scope. This is exactly what Ben blogged about in January last year.

In the next part, I’ll take a look at scopes.

Cheers,
John.

Windows Server 2008 R2 RTM on Technet & MSDN

John Howard -MSFT — Fri, 14 Aug 2009 21:38:00 GMT

After the Windows 7 RTM release was published last week, it was joined by Windows Server 2008 R2 RTM for Technet and MSDN subscribers this morning. Woo hoo!

Have fun with Hyper-V and all the great new features and improvements.

Cheers,
John.

Windows 7 RSAT final build available for download

John Howard -MSFT — Wed, 12 Aug 2009 09:25:00 GMT

The Remote Server Administration Tools (RSAT) for Windows 7 RTM have been released to microsoft.com. These include the Hyper-V tools for remote management of Windows Server 2008 R2 and Microsoft Hyper-V Server 2008 R2.

You can download them from http://www.microsoft.com/downloads/details.aspx?FamilyID=7d2f6ad7-656b-4313-a005-4e344e43997d&displaylang=en – remember to download the right architecture (x86 or x64) as necessary.

Once you install the update, you can enable one or more of the following tools from the Control Panel under Programs, Turn Windows features on or off.

Server Administration Tools:

Server Manager

Role Administration Tools:

Active Directory Certificate Services (AD CS) Tools
Active Directory Domain Services (AD DS) Tools
Active Directory Lightweight Directory Services (AD LDS) Tools
DHCP Server Tools
DNS Server Tools
File Services Tools
Hyper-V Tools
Terminal Services Tools

Feature Administration Tools:

BitLocker Password Recovery Viewer
Failover Clustering Tools
Group Policy Management Tools
Network Load Balancing Tools
SMTP Server Tools
Storage Explorer Tools
Storage Manager for SANs Tools
Windows System Resource Manager Tools

Cheers,
John.

HVRemote refresh

John Howard -MSFT — Sat, 08 Aug 2009 02:21:00 GMT

So after some 22,000 downloads (thank you) in the 8 months since first released, HVRemote has undergone a refresh to make it even easier to configure Hyper-V Remote Management and diagnose issues.

The major change in version 0.7 is the ability to perform some verification of the configuration and provide hints as to what to follow if it detects an error through the use of the new /target option. Below is an example of the new bit of output running hvremote /show /target:servername from a Windows 7 client where everything is working just fine (yes, it is fine for the ping to timeout, that just means it’s blocked by the firewall).

HVRemote 0.7 fully supports Windows Server 2008 R2, Microsoft Hyper-V 2008 R2 and Windows 7. In addition, the home page has undergone a refresh to cover the some of the most commonly asked questions, and the documentation has been brought up to date.

For a list of other changes, please see the documentation.

Cheers,
John.

Windows Virtual PC Links

John Howard -MSFT — Fri, 07 Aug 2009 06:45:00 GMT

A quick note to say that Windows Virtual PC has reached RC (release candidate) and is available for download. Ben has lots more information here, and I'm really pleased to see that Prasad has started a WVPC team blog where you can learn more about its features and capabilities.

Cheers,
John.