Amit Pawar - Infrastructure blog

Microsoft.com Powered by Hyper-V

Now that Hyper-V has been around for a month or so I keep getting asked for early proof points and case studies of where Hyper-V is being used in Production.

The case study that really stands out in my mind is an internal one at Microsoft. One of the more challenging systems from a server subsystem utilisation perspective is www.microsoft.com. The site handles 15,000 requests per second, 1.2 billion page views per month, and 280M worldwide unique users per month as well as supporting ~5000 content contributors from within the company. This site has close to 300GB of content consisting of some seven million individual files on each server. Due to this scale and the variety of applications hosted, the site heavily exercises all of the major subsystems - memory, CPU, network, and file I/O – on each server. Based on the load characteristics and the fact that this site is a testing ground for early adoption of Microsoft technology, they expected the production load of www.microsoft.com to provide a great test for Hyper-V.

On June 5^th the Operations Team turned up a full sixteen VM cluster hosting www.microsoft.com on Hyper-V. That cluster is handling 25% of the production traffic load and can scale past that to support data center redundancy goals. Microsoft.com has not encountered any performance, stability or availability issues on the virtualised cluster.

The Deployment

The team began the deployment on the www.microsoft.com site with a single server back in March running on Hyper-V Beta.  They continued with live load testing and artificial stress load similar to the approach with the MSDN deployment. This single VM was as stable, reliable and performed better with live internet load as compared to the older physical servers in the cluster. With the success of the first VM running www.microsoft.com they decided to expand to an entire cluster of servers. This was also a great opportunity to leverage SCVMM 2008 beta for the first time in production. 

At the time the SCVMM 2008 beta required Hyper-V RC0 so in order to use SCVMM and Hyper-V together, RC0 was utilised through the deployment phase. Once the deployment was complete, the servers were all upgraded to Hyper-V RC1.

Using SCVMM they created a “golden” web server image for www.microsoft.com including both the server and content to improve deployment speed as well as configuration control. Previously a new deployment of www.microsoft.com involved 12 hours to sync the 7 million small content files over the network. Utilizing a single content VHD cut this time down to 4 hours.

Limited test hardware available for this first phase so, were only able to deploy one VM per physical server. Clearly this is not an optimal strategy for long term virtualisation given that each server has 8 processors, but it did allow us to move quickly with the hardware they had available. The next stage of the www.microsoft.com virtualization will take place on a SAN based infrastructure allowing us to run multiple VMs per server head.

Current www.microsoft.com Virtualised Environment

Availability

One of the primary goals is maintaining high availability regardless of where they are in the technology lifecycle. They measure availability in a variety of ways, but one of the baseline tests they use is a 3^rd party provided HTTP request from 45 worldwide agents against the www.microsoft.com hosting platform – currently Windows Server 2008, IIS7 and now Hyper-V.   The average availability of the platform prior to the Hyper-V based deployment was 99.94% and is running at an average of 99.95% since the deployment of the first cluster.  Since this particular measure is an Internet based test, meeting or exceeding previous results means they hit their goals.

Platform availability before and after Hyper-V Deployed to handle 25% of traffic

Performance

The team at Microsoft.com has been very encouraged by the stability, scalability and performance of Hyper-V on the www.microsoft.com site.  In terms of performance for this site, overall the results are in-line with previously observed measures while virtualizing MSDN and TechNet. As with those sites they completed comparison testing of the VMs against both the current and new physical servers. The outcome of the current physical servers vs. new VM comparison helped us determine how many VMs running www.microsoft.com they would need to match the current physical server capacity as well as handle projected growth. Given the VM performance on the new servers we’ll consolidate down from 80 physical servers to 64 VMs. Those VMs will initially be deployed onto a total of 40 new physical servers.

The initial performance testing showed a 10% CPU overhead in running www.microsoft.com in a virtual machine. This testing was based on sustained live traffic using matching hardware for the VM host and the physical server. Both the physical server and the VM were configured with four processors, 30GB RAM and included matching disk and network subsystems to provide for an accurate comparison.

Based on these results they are ready to fully host www.microsoft.com web servers on Hyper-V and we’re targeting end of June for 50% of the load. As soon as they complete deployment of the new hardware infrastructure in diverse data centers, we’ll complete the full virtualization.

If you would like to see me try to speak to all of this, and about our team’s overall adoption success with Hyper-V in under five minutes you can find a short video noted below.

Also check out our TechCenter for further information about our group’s technology adoption efforts.

I believe overall this is a great way of approaching the virtualsation of a core workload. I would be interested in reading about your experience with Hyper-V.