Windows Server Division WebLog

Hi—I am Rob Emanuel, a Technology Architect on the Microsoft.com Operations team focusing on virtualization.  I wanted to share the great progress we have made rolling out Hyper-V since my first blog a month ago.   In that blog I discussed our success with fully virtualizing the web front ends for MSDN and TechNet on Hyper-V RC0.   I also highlighted an article our team wrote about how we approached those virtualizations and hopefully conveyed how truly successful we found Hyper-V to be as a web platform.

Microsoft.com Powered by Hyper-V

One of our more challenging systems from a server subsystem utilization perspective is www.microsoft.com.  The site handles 15,000 requests per second, 1.2 billion page views per month, and 280M worldwide unique users per month as well as supporting ~5000 content contributors from within the company.  This site has close to 300GB of content consisting of some seven million individual files on each server.  Due to this scale and the variety of applications hosted, the site heavily exercises all of the major subsystems - memory, CPU, network, and file I/O – on each server.  Based on the load characteristics and the fact that this site is a testing ground for early adoption of Microsoft technology, we expected the production load of www.microsoft.com to provide a great test for Hyper-V.

On June 5^th our Operations Team turned up a full sixteen VM cluster hosting www.microsoft.com on Hyper-V.   That cluster is handling 25% of the production traffic load and can scale past that to support data center redundancy goals.  We have not encountered any performance, stability or availability issues on the virtualized cluster. 

The Deployment

We began our deployment on the www.microsoft.com site with a single server back in March running on Hyper-V Beta.  We continued with live load testing and artificial stress load similar to our approach with the MSDN deployment.  This single VM was as stable, reliable and performed better with live internet load as compared to the older physical servers in the cluster.   With the success of the first VM running www.microsoft.com we decided to expand to an entire cluster of servers.  This was also a great opportunity to leverage SCVMM 2008 beta for the first time in production. 

At the time the SCVMM 2008 beta required Hyper-V RC0 so in order to use SCVMM and Hyper-V together, RC0 was utilized through the deployment phase.  Once the deployment was complete, the servers were all upgraded to Hyper-V RC1.  

Using SCVMM we created a “golden” web server image for www.microsoft.com including both the server and content to improve deployment speed as well as configuration control.  Previously a new deployment of www.microsoft.com involved 12 hours to sync the 7 million small content files over the network.  Utilizing a single content VHD cut this time down to 4 hours. 

We had limited test hardware available for this first phase so, were only able to deploy one VM per physical server.  Clearly this is not an optimal strategy for long term virtualization given that each server has 8 processors, but it did allow us to move quickly with the hardware we had available.  The next stage of our www.microsoft.com virtualization will take place on a SAN based infrastructure allowing us to run multiple VMs per server head. 

Current www.microsoft.com Virtualized Environment

Availability

One of our primary goals is maintaining high availability regardless of where we are in the technology lifecycle.  We measure availability in a variety of ways, but one of the baseline tests we use is a 3^rd party provided HTTP request from 45 worldwide agents against the www.microsoft.com hosting platform – currently Windows Server 2008, IIS7 and now Hyper-V.   The average availability of the platform prior to our Hyper-V based deployment was 99.94% and is running at an average of 99.95% since the deployment of the first cluster.  Since this particular measure is an Internet based test, meeting or exceeding previous results means we’ve hit our goal.

Platform availability before and after Hyper-V Deployed to handle 25% of traffic

Performance

We have been very encouraged by the stability, scalability and performance of Hyper-V on the www.microsoft.com site.  In terms of performance for this site, overall the results are in-line with previously observed measures while virtualizing MSDN and TechNet.  As with those sites we completed comparison testing of the VMs against both the current and new physical servers.  The outcome of the current physical servers vs. new VM comparison helped us determine how many VMs running www.microsoft.com we would need to match the current physical server capacity as well as handle projected growth.   Given the VM performance on the new servers we’ll consolidate down from 80 physical servers to 64 VMs.   Those VMs will initially be deployed onto a total of 40 new physical servers. 

Our initial performance testing showed a 10% CPU overhead in running www.microsoft.com in a virtual machine.   This testing was based on sustained live traffic using matching hardware for the VM host and the physical server.   Both the physical server and the VM were configured with four processors, 30GB RAM and included matching disk and network subsystems to provide for an accurate comparison.  

Based on these results we are ready to fully host www.microsoft.com web servers on Hyper-V and we’re targeting end of June for 50% of the load.  As soon as we complete deployment of our new hardware infrastructure in diverse data centers, we’ll complete the full virtualization. 

Also check out our TechCenter for further information about our group’s technology adoption efforts.

I hope you enjoy virtualizing on Hyper-V as much as we have.

-Rob