August, 2009

I now have the final list of sessions that I will be attending at VMworld. As I noted last time I had a lot of waitlists which it now appears I have not made the cut for most of the sessions. That means I will be at our booth (#2422) almost all of today (Aug 31, 2009) and at the times when I am not attending a session.

There is a great opportunity to win a 32GB Zune HD. All you have to do is to follow the Microsoft Virtualization experts on Twitter. By the way this is open to folks who are not attending VMworld as well. So follow along with us!! You can follow me here.

I am heading over to the Moscone Center to register and see if I can get into some of the labs being offered today.

Vijay Tewari

Principal Program Manager, Windows Server Virtualization

We are really excited to announce that Microsoft Hyper-V Server 2008 R2 is available for download here. It’s been a long an exciting journey bringing this product to our customers.

The chart below tells it all, we have listened to our customers and have designed the product to meet their needs. This is a robust virtualization platform with live migration built in at no extra charge.

Comparing Hyper-V Server V1 vs. V2

And don’t forget that Microsoft Hyper-V Server 2008 R2 also supports boot from flash for our OEM partners.

Just to recap this release has the following features

(a)    High availability and live migration for managing a dynamic IT infrastructure

(b)    Support for 64 logical processors future proofing our customers to scale up with the hardware

(c)    Support for running up to 384 virtual machines with up to 512 virtual processors

(d)    Processor compatibility mode for live migration across different processor SKU’s from the same vendor

(e)    Hot add/remove virtual storage

(f)     Networking enhancements (VMQ, Chimney, support for Jumbo Frames)

(g)    Simplified management using sconfig

(h)    Boot from flash

We will be at VMworld to meet customers and share this news with them. Stop by our booth (#2422) and follow us as we blog and tweet.

Vijay Tewari

Principal Program Manager, Windows Virtualization Team

This post is for the readers out there that will be attending VMworld, or have colleagues attending VMworld.

There’s been some speculative statements made in the press and blogs in the past 24 hours, so I want to try to set the record straight:

1.       Microsoft will be exhibiting at VMworld 2009. Can't wait! We have a 10x10 booth, #2422. If you have a chance, please stop by booth. It's right next to the Blogger lounge. See more below.

2.       Unlike prior VMworld conferences, we’re no longer allowed to sponsor the event. We can only be an exhibitor. Why? Let's look at excerpts from VMworld’s rules (my highlights):

VMworld 2009 Sponsor and Exhibitor Rules and Regulations

Booth/Session Demonstrations (pg. 2)

Sponsors and exhibitors must market or demonstrate products on the exhibition floor and in the sessions

which are complementary to VMware products and technologies. Complementary products and services are

defined as products/services that do not overlap/substitute with VMware’s products/capabilities, and help

expand the reach and solution scope of VMware’s capabilities solely as deemed by VMware. All sponsors

and exhibitors must adhere to following guidelines in regards to booth demonstrations (please also review

the “Promotion, Giveaways, Contests” section below):

Eligibility (pg. 4)

To sponsor or exhibit at VMworld a company must be a VMware partner in good standing in our TAP,

Strategic Global Partner or VIP Partner Programs. Companies that are not VMware partners may be allowed

to exhibit at VMworld under exception by VMware.

Both sides (us and VMware) agreed that, apart from the features in System Center Virtual Machine Manager that manage ESX, Microsoft products aren’t complementary to VMware’s products. We discussed and confirmed their intent of these clauses in early August, and had a follow-up Aug. 11 email from VMware, that reads in part:

As I mentioned, the rules require all exhibitors to market and demonstrate products that are complimentary to VMware products and technologies.  We are open to reviewing any proposals for complimentary positioning of Microsoft products and demos at the show…

As a result, based on the clause above, phone/email with VMware employees, we don't believe we have the right to demo our products in the booth. This decision runs counter to Microsoft’s geek culture, as you can imagine, but we've also become more pragmatic over the years :-). 

So what is Microsoft doing at VMworld? 

We’ve learned over the course of 4 years attending VMworld (first was Vegas 2005) that there are many attendees who use, admin, manage, sell, support Windows Server, SQL Server, Exchange Server, Windows XP/Vista, System Center, etc. They want the opportunity to engage and receive information from Microsoft virtualization experts. So we’ll have Microsoft virtualization experts in the booth. These experts will be there to answer your questions. They’ll also be Tweeting about their experiences and impressions as they talk to customers and attend sessions. I encourage you to follow them on Twitter -  see the table below for their usernames.

We’ll also have two customers in the booth at different times. I’ll be there. See you there!

Patrick O’Rourke

Hi, I am Vijay Tewari a program manager with the Virtualization team here at Microsoft. Having spent the last 15 months or so deeply involved in the development of Hyper-V I am really looking forward to meeting customers and partners at VMworld 2009.

Much is being made about the fact that we are not showing Hyper-V and System Center Virtual Machine Manager at this event due to the show policies. I will stay away from that debate although coming on the heels of our RTM I would have loved to show the product. However, a lot of us will be there at the event and are really looking forward to interacting with customers. At the end of the day the product is only useful if it meets our customers’ demands and there is nothing like having a face to face interaction with our customers to hear firsthand what they like and hate about the product.

Below is my tentative schedule for VMworld. As you can see a lot of wait lists, but I am hoping that I can squeeze in. Would like your feedback if there are sessions you would like to see covered.

We will be tweeting (follow vtango on Twitter) and blogging all through the event, so follow along.

Looking forward to seeing you!!

Vijay

Vijay Tewari, Principal Program Manager, Virtualization

Ordinarily it'd be great news that we RTM'd System Center Virtual Machine Manager 2008 R2 ahead of schedule, so close to the RTM of WS08 R2 Hyper-V and one week before VMworld conference. You can't beat the timing of it all. And the ~10,000 beta testers of SCVMM08 R2 are a big part of why we were able to deliver a high-quality release in the time we did.  Some of the early adopters included Continental Airlines, Lionbridge Techologies, Sporton, Indiana University Auxilary IT and MaximumASP. Thanks to all!

But unfortunately, next week at VMworld 2009 we can't show SCVMM 2008 R2, or any other products, in our booth. You've probably read about it. In short, it's their show, it's not an industry show, and they set the rules. So we'll make the best of it; always lots of Microsoft customers and partners on the floor. Stop by the booth to meet some of the best/brightest minds at Microsoft.

As for the RTM of SCVMM 2008 R2, Zane summarizes his thoughts here. Kerim blogged about upgrade path to R2 from SCVMM 2008. See here. And on May 11, Vishhwa provided some details on features of SCVMM 2008 R2 here.

As Cheng wrote in his blog (I can't say it any better)

Go download a trial and give it a try, go get it now, and unleash the virtualization power that Microsoft brings to you.

Patrick

Hi all,

My name is Dave Bermingham and I’m the director of Windows product management for SteelEye Technology, a provider of business continuity and disaster recovery solutions. I’d like to thank Microsoft’s Virtualization team for the opportunity to guest blog on SteelEye’s continued support for Hyper-V as well as exciting new features coming in support of Hyper-V on Windows Server 2008 R2. 

I can’t believe it is almost a year ago since I demonstrated SteelEye DataKeeper Cluster Edition at the Microsoft Virtualization Launch event in Bellevue, WA.  In the past year we have seen the interest in our Hyper-V disaster recovery products increase dramatically and with the recent release of Windows Server 2008 R2, I believe we have just seen the tip of the iceberg!

SteelEye’s mission is to “Replicate Any Data, Protect Any Application.” We pride ourselves on our commitment to provide the Windows marketplace with flexible, scalable and cost-effective solutions that are enterprise-grade yet customizable for any size business.  Most recently SteelEye identified an opportunity to extend Microsoft’s Windows Server 2008 Failover Clustering (WSFC) by delivering SteelEye DataKeeper Cluster Edition, a real-time data replication solution that integrates seamlessly with WSFC to enable multi-site clusters while eliminating the single point of failure traditionally associated with shared storage resources.  A comprehensive and cost-effective disaster recovery solution, DataKeeper Cluster Edition helps supports improved productivity and availability of your Hyper-V VMs. 

SteelEye soon will be announcing support for Windows Server 2008 R2 and the highly anticipated “Live Migration”.  As our Windows team continues to develop and build additional capabilities for enhanced disaster recovery support of Hyper-V, we welcome your comments and suggestions. To learn more about the latest developments for Hyper-V support, take a look at our FAQs or visit the SteelEye DataKeeper Cluster Edition page on our website.

Thanks,

Dave

Virtualization Nation,

In my last blog, I discussed the importance of a Security Development Lifecycle and how we're making resources available to our customers and partners. Today, I want to discuss a more resources Microsoft provides our partners to develop and refine their products running on Windows and point out opportunities for integration with Hyper-V. There are a number of ways we work with partners to improve code quality and security starting with Windows Error Reporting.

Windows Error Reporting (WER)

Windows Error Reporting is a set of Windows technologies that capture software crash data and support end-user reporting of crash information. Through Windows Quality Online Services (Winqual), software and hardware vendors can access reports in order to analyze and respond to these problems. WER technologies are implemented in Windows XP, Vista, Windows 7, Windows Server 2003/2003 R2 and Windows Server 2008/2008 R2.

Vendors can use WER to view error reports at no charge. This service is available for all products, even those that do not qualify for the "Designed for Windows" logo-although we strongly recommend that you submit your products to the Windows Logo Program.

With data from Microsoft Windows Error Reporting (WER), you can identify the most common real-world customer problems and quickly provide a solution to your customers. While customer support calls provide information about common issues, they do not always provide enough detail to debug the actual code. Further, support records indicate only those problems which prompted calls. Because it is much easier for users to submit an error report than to contact customer support, WER can provide a more comprehensive picture of the problems your customers are having. And because WER collects data at the point of failure, you can get a more detailed picture of what is going wrong.

In the end, this is a win-win situation. PARTNERS WIN because they can diagnose and produce better code (and lower support costs) and CUSTOMERS WIN with higher quality solutions. If your company develops software and aren't using WER, you need to take advantage of this resource. Here's the link to the WER site:

http://www.microsoft.com/whdc/winlogo/maintain/StartWER.mspx

While WER can be a boon to developers by providing critical information to triage issues, we didn't stop there. We also released a new tool, called the !exploitable Crash Analyzer, on CodePlex March 20 at the CanSecWest conference in Vancouver, British Columbia. A Windows debugger extension, this heuristics-based tool is aimed at not only helping developers assess what is causing crashes, but also ranking the seriousness of a bug.

Introducing !exploitable Crash Analyzer: What does it do?

The program works by examining crash data-information gathered when an application stops performing its expected function-to identify the unique issues that caused the crash. From there, the program provides guidance on how exploitable the crash is, and can be used by third-party developers to then prioritize the problem.

"As a tool, it can save developers time and effort," said Roger Kay, president of tech industry analyst group Endpoint Technologies Associates. "A number of apparently different crashes can actually be caused by the same code. The analyzer isolates the offending block and essentially says, 'Here, all these different crashes are actually the same failure, and it's an important one that you ought to fix right away because it presents an open attack surface.'"

If you're interested in this tool, go to the Microsoft Security Engineering Center where you can download the tool from CodePlex (CodePlex is Microsoft's open source project hosting web site) and find how this can help in your development.

The link is here: http://www.microsoft.com/security/msec/default.mspx

Cheers,

Jeff Woolsey

Principal Group Program Manager

Windows Server, Hyper-V

In my last two blog posts (Part 1 & Part 2), I started an in depth analysis tackling VMware's claims head on that because their disk footprint is smaller and ESX/ESXi are single purpose hypervisors, they are therefore more secure. If that's the case, then it stands to reason that ESX/ESXi:

• should have fewer patches (they have less code to patch)
• patches should be smaller in disk footprint (they have a smaller codebase and you want to keep code churn to a minimum; otherwise one could ship a 1k stub file and claim to be smaller)
• should offer higher availability, reliability and uptime

Using VMware's own metrics:

• In part 1, Microsoft Hyper-V Server 2008 clearly won over VMware ESXi 3.5
• In part 2, Windows Server 2008 Hyper-V clearly won over VMware ESX 3.5

In part 3, lets take a look at VMware's favorite comparison Windows Server 2008 Hyper-V to ESXi 3.5. Let's have a look.

Stacking The Deck In VMware's Favor

In this last comparison, I will freely admit that this isn't an apples to apples comparison. In this comparison, I gave ESXi 3.5 a 6 month advantage. Here's what I mean. Specifically, I compare:

• VMware ESXi 3.5 from June 30th 2008 to June 30th 2009 (a 12 month period)
• Windows Server 2008 Hyper-V January 1 2008 to June 30th 2009 (an 18 month period)

Using an 18 month sample set for Windows Server 2008 covers the majority of its time in market and goes to the heart VMware's fundamental claim that because their disk footprint is smaller and ESX/ESXi are single purpose hypervisors, they are therefore more secure. This tilts the scale in VMware's favor.

Windows Server 2008 Hyper-V to VMware ESXi 3.5

Disk Footprint & Patch Count. Here's what we found:

• Windows Server 2008 Full Installation: 32 patches totaling 408 MB of patches
• Windows Server 2008 Core Installation: 26 patches totaling 82 MB of patches or (~20% fewer than a Windows Server 2008 full installation)
• VMware ESXi 3.5: 13 patches, totaling over 2.7 GB.

Yes, I said over 2.7 GB. To put it another way,

• VMware ESXi 3.5 had a 6.6x greater patch footprint than Windows Server 2008 (Full)
• VMware ESXi 3.5 had a 33x greater patch footprint than Windows Server 2008 (Core)

So much for the disk footprint argument. Again, how can the ESXi footprint be so huge?

Because VMware releases a whole new ESXi image every time they release a patch. Furthermore, because VMware releases a whole new ESXi image every time they release a patch it also means that every ESXi 3.5 server requires a reboot. At this point, a VMware salesman may actually concede that every ESXi server has to be rebooted for every patch, but they will then state that they have VMotion (Live Migration) so it doesn't affect their uptime.

Except when their own patches cause days of downtime and render VMotion impotent.

Reliability/Availability. With VMware ESXi 3.5 Update 2, it included a serious flaw which resulted in two days of downtime for their customers including the loss of VMotion:

"Starting this morning, we could not power on nor VMotion any of our virtual machines," said someone identified as "mattjk" on a VMware support forum. "The VI Client threw the error 'A general system error occurred: Internal Error.'"

It was so bad, VMware's CEO had to apologize on numerous occasions. (HERE, HERE, HERE, HERE, HERE, & HERE). VMware then rushed out the VMware ESXi 3.5 Update 3 which introduced instability to VMware High Availability and could cause virtual machines to spontaneously reboot. (HERE & HERE)

Virtual machines that spontaneously reboot due to bugs in VMware high availability.

Now consider the fact that there were two significant quality and reliability issues with two major updates in a row (ESX/ESXi Update 2 & Update 3). While the initial Windows Server 2008 Hyper-V release didn't provide Live Migration (Windows Server 2008 Hyper-V R1 had Quick Migration and Windows Server 2008 Hyper-V R2 includes Live Migration for free), it didn't include two days of potential downtime and virtual machines unexpectedly rebooting either. For those that track availability in terms of nines (five nines is 5.26 minutes of downtime a year) VMware Update 3.5 Update 2 dropped customers to "two nines" of availability.

Using VMware's own metrics, Windows Server 2008 Hyper-V is clearly the winner over ESXi 3.5.

The Facts Contradict VMware's Claims

As stated at the beginning of this series, VMware's overarching point is because their disk footprint is smaller and ESX/ESXi is a single purpose hypervisor, they are therefore more secure. While VMware heavily touts this claim (it's in numerous location on their website for starters), the facts from this analysis directly contradict their claims. Specifically:

1. The platform with the largest number of patches was VMware ESX 3.5 with 85 Security & Critical Patches averaging over a patch per week.
2. The platform with the largest patch footprint was VMware ESX 3.5 totaling over 3 GB worth of patches followed by VMware ESXi 3.5 with over 2.7 GB. That's right, VMware's single purpose virtualization platforms that claims to have the smallest footprint had the two largest patch footprints by about a mile. (Graph below)
3. Both VMware ESX & ESXi had a recent case of the most severe virtualization flaw with guest code able to break out of the virtual machine and could potentially:
   • Provides administrator access, Allows complete confidentiality, integrity, and availability violation; Allows unauthorized disclosure of information; Allows disruption of service.
4. VMotion/Live Migration is not a panacea to patching. It can help, but in the case of VMware's own self-inflicted faulty patch, it rendered their advantage impotent.
5. VMware had not just one, but two significant updates with serious quality and reliability issues with both ESX and ESXi. Specifically, ESX/ESXi Update 2 Issues: HERE, HERE, HERE, HERE, HERE, & HERE & ESX/ESXi Update 3 Issues: HERE & HERE.

The Point Of This Series

Say it with me:

>  Security is more than just disk footprint. <

Quoting disk footprint size alone is a nice pithy, superficial phrase, but it's also a boat load of bollocks. The next time some VMware representative throws out that argument, point them to this blog and tell them Jeff sent you. If you've ever spent anytime with a security expert, one of the first things they will tell you is that security is not a one time exercise. Security is an ongoing process that should be embedded throughout the entire development lifecycle. It's that belief that drove us to develop the Microsoft Secure Development Lifecycle (SDL) and is publicly available.

Microsoft Secure Development Lifecycle (SDL)

The concepts that make up the Microsoft SDL were formed with the Trustworthy Computing (TwC) directive of January 2002. At that time, many software development groups at Microsoft instigated "security pushes" to find ways to improve the security of existing code.

Becoming a mandatory policy in 2004, the SDL represents a major cultural evolution at Microsoft with regards to software security and privacy and has matured into a well defined methodology. A "security process by a software company," the SDL was designed as an integral part of the development process. The development, implementation and constant improvement of the SDL represents a strategic investment for Microsoft, and an evolution in the way that software is designed, developed, and tested.

From a high level, the Microsoft SDL looks like this:

The Microsoft Security Development Lifecycle

Benefits of the Microsoft SDL:

• Reducing the number of software vulnerabilities

  The SDL has played a critical role in embedding security and privacy into Microsoft software and culture, leading to measurable and widely recognized security improvements in flagship products such as Windows and SQL Server and the proof is real. How about:

  • Windows XP to Vista a 45% decrease

  • SQL Server 2000 to 2005 91% decrease

• Windows Server 2008 Full vs Server Core
  • Reduction in patches by ~50%
• Reducing the total cost of development

  The SDL reduces the "total cost of development" by finding and eliminating vulnerabilities early. According to the National Institute of Standards and Technology (NIST), eliminating vulnerabilities in the design stage can cost 30 times less than fixing them post release.

Read that last sentence again.

Thirty times.

I also want to point out that the Microsoft Security Development Lifecycle doesn't end once the bits are released. It also means having a well-established response mechanism including:

• responding to potential security threats
• root cause analysis to understand why the issue occurred and ensure that issue isn't repeated
• issuing security patches

The importance of a security development lifecycle cannot be understated. No matter how well you execute, there is no such thing as perfect code. Whether it's Microsoft, VMware, <insert software vendor here>, having a rigorous security development practices in place is imperative. And, in case you think I'm satisfied with our patch numbers above, you'd be wrong. I don't ever want to get complacent and think for a moment that "security is done." Security is never done.

Let he who has written perfect code throw the first stone.

Let's Have A Look At The VMware's Security Development Lifecycle

So, where's the VMware Security Development Lifecycle?

You're guess is as good as mine.

I went to VMware's site and searched for their security development lifecycle. I found their Security Center which lists their patches, but that's just one small aspect to a security development lifecycle. I Bing searched "VMware security development lifecycle" and was returned a sales pitch from VMware to buy something at $1500 per processor.

No, I'm not kidding.

Making The SDL Available To Our Partners

After a significant investment in time, money, manpower we've developed and want to give back to our partners. A great place to start is the Microsoft SDL Homepage. Here you will find whitepapers, best practices, threat modeling tools, process guidance and much more. In addition, we recently released the Microsoft SDL Process Template for Visual Studio Team Systems. This template helps ease the adoption of the SDL, demonstrates security return on investment and provides auditable security requirements and status.

I'd be remiss if I didn't point out an excellent book aptly titled, Writing Secure Code Vol. 2 and point to the blog of one of the authors, Michael Howard. More links below.

Microsoft SDL Homepage: http://msdn.microsoft.com/en-us/security/cc448177.aspx

Microsoft SDL Process Template for Visual Studio Team System: http://msdn.microsoft.com/en-us/security/dd670265.aspx

Writing Secure Code Volume 2: http://www.amazon.com/Writing-Secure-Second-Michael-Howard/dp/0735617228

Michael Howard's Blog: http://blogs.msdn.com/michael_howard/

In my next blog, we'll discuss more free tools and programs available our partners.

Cheers,

Jeff Woolsey

Principal Group Program Manager

Windows Server, Hyper-V

Virtualization Nation,

In part 1 of this series, I started an in depth analysis tackling VMware’s claims head on that because their disk footprint is smaller and ESX/ESXi are single purpose hypervisors, they are therefore more secure.

The Thin Hypervisor Debate In A Nutshell

Rather than attempt to restate VMware's argument, let's just use their exact verbiage. From VMware's website:

It's interesting to point out that VMware uses ESX & ESXi 3.5 in the column title, but then only uses ESXi for comparison in the right hand column and then rushes to compares ESXi to Windows Server 2008 Hyper-V.

Gosh, I wonder why ESX 3.5 isn't included? We'll get to that in a sec.

1. Disk footprint. Disk footprint. Disk footprint. They're quite fixated on that.
2. Reduce the risk of downtime due to fewer patches (i.e. fewer patches is good)
3. Increase reliability (and, in turn, availability)

VMware's overarching point is because their disk footprint is smaller and ESX/ESXi is a single purpose hypervisor, they are therefore more secure. If that's the case, then it stands to reason that ESX/ESXi:

• should have fewer patches (they have less code to patch)
• patches should be smaller in disk footprint (they have a smaller codebase and you want to keep code churn to a minimum; otherwise one could ship a 1k stub file and claim to be smaller)
• should offer higher availability, reliability and uptime

We're going to put that to the test.

Time For Some Analysis

With VMware's own metrics in mind, I decided we should perform an apples to apples comparison. Specifically, over an 18 month period from January 2008 to June 30th 2009, comparing Windows Server 2008 (R1) and VMware ESX 3.5 in terms of numbers of patches, size of patches and availability. I specifically chose these versions because I wanted to take a reasonable, historical sample size (18 months) of both platforms. Going back 18 months ensures the sample set covers the majority of Windows Server 2008 time in market and tackles VMware’s fundamental claim that because their disk footprint is smaller and ESX/ESXi are single purpose hypervisors, they are therefore more secure. If anything this should work in VMware’s favor. Finally, I didn’t use Hyper-V R2 and VSphere because they both RTM'd within the last 90 days and that wouldn't be statistically relevant.

What did we compare? Wanting to keep the comparisons as close as possible, here's what we analyzed:

• VMware: Security and Critical Patches
• Microsoft: Critical and Important Patches

Both Microsoft and VMware also have optional patches, but I didn't include this type because they are optional, so we could focus on the most acute patches. Finally, let me very clear about the Microsoft patches. These counts include ALL critical and important patches, meaning if there was a critical patch for IE or some other Windows component I counted it. Had I ignored these types of patches, that wouldn’t be a fair comparison.

Comparison #2: Windows Server 2008 Hyper-V & VMware ESX 3.5

Remember earlier how I thought it was interesting that VMware likes to use ESXi in its comparisons and that ESX is conspicuously absent?

Now I know why.

Let’s take a look shall we.

Disk Footprint & Patch Count: Here’s what we found:

• Windows Server 2008 Hyper-V (Full) Installation: 32 patches totaling 408 MB of patches
• Windows Server 2008 Hyper-V (Core) Installation: 26 patches totaling 82 MB of patches or ~20% fewer than a full installation
• VMware ESX 3.5: 85 patches totaling over 3 GB

Yes, that’s right 85 patches totaling over 3 GB . To put it a few other ways, 

• VMware ESX had 2.6x greater number of patches than Windows Server 2008 (Full)
• VMware ESX had 7.3x greater patch footprint than Windows Server 2008 (Full)
• VMware ESX had 3.2x greater number of patches than Windows Server 2008 (Core)
• VMware ESX had 36x greater patch footprint than Windows Server 2008 (Core)

The News Gets Worse

In addition, VMware ESX averaged over a patch per week for 18 months with:

• 46 Critical patches (1.4 GB)
• 39 Security patches (1.6 GB) including

Guest code able to break out of VM and into the ESX Hypervisor

For those you who may not know, code running in a guest operating system that is able to break out of a VM and into a hypervisor is THE WORST type of security flaw you can have for a virtualization platform. Period. VMware has had exploits on both their “thin hypervisor (ESXi)” and on their “thick hypervisor (ESX)” as recently as a few months ago.

For example, April 2009 CVE-2009-1244: A critical vulnerability in the virtual machine display function allows a guest operating system users to execute arbitrary code on the host OS.

Impact: Provides administrator access, Allows complete confidentiality, integrity, and availability violation; Allows unauthorized disclosure of information; Allows disruption of service.

Something to consider when VMware wants to host your data in “the cloud.”

(BTW: As mentioned above, I tried to keep this an apples to apples comparison and didn’t include VMware’s “General” Patches. Had I included those as well, that would have added another 83 general patches and another 1 GB of patch footprint, but I digress…)

In addition, don’t forget this one…

Reliability/Availability. With VMware ESX 3.5 Update 2, it included a serious flaw which resulted in two days of downtime for their customers including the loss of VMotion:

"Starting this morning, we could not power on nor VMotion any of our virtual machines," said someone identified as "mattjk" on a VMware support forum. "The VI Client threw the error 'A general system error occurred: Internal Error.'"

It was so bad, VMware’s CEO had to apologize on numerous occasions. (HERE, HERE, HERE, HERE, HERE, & HERE). VMware then rushed out the VMware ESX 3.5 Update 3 which introduced instability to VMware High Availability and could cause virtual machines to spontaneously reboot. (HERE & HERE)

Virtual machines that spontaneously reboot due to bugs in VMware high availability.

Now consider the fact that there were two significant quality and reliability issues with two major updates in a row (ESX/ESXi Update 2 & Update 3). While the initial Windows Server 2008 Hyper-V release didn’t provide Live Migration (Windows Server 2008 Hyper-V R1 had Quick Migration and Windows Server 2008 Hyper-V R2 includes Live Migration for free), it didn’t include two days of potential downtime and virtual machines unexpectedly rebooting either. For those that track availability in terms of nines (five nines is 5.26 minutes of downtime a year) VMware ESX 3.5 Update 2 dropped customers to “two nines” of availability.

Using VMware’s own metrics, Windows Server 2008 Hyper-V is clearly the winner over ESX 3.5 and by a pretty wide margin.

In my next blog, we’ll analyze VMware’s favorite comparison, Windows Server 2008 Hyper-V and VMware ESXi 3.5.

Cheers,

Jeff Woolsey

Principal Group Program Manager

Windows Server, Hyper-V

Virtualization Nation,

In my last blog post, I started an in depth analysis tackling VMware’s claims head on that because their disk footprint is smaller and ESX/ESXi are single purpose hypervisors, they are therefore more secure. I read some posts on the blogosphere that had questions about the methodology. Let’s dive in.

Didn’t Compare The Number Of Reboots

You’re right I didn’t. I didn’t think comparing the number of reboots was needed after:

• establishing that VMware’s own patches caused customers two days of downtime without the use of VMotion to mitigate that downtime
• referring to VMware’s own updates that caused VMs to spontaneously reboot due to bugs in VMware HA
• pointing out that every ESXi patch requires a reboot

Even if every Microsoft Hyper-V Server 2008 patch came out on a different day (that wasn’t the case most of the patches came out in groups) and required a reboot at the time, there’s no way on earth that the cumulative time would be in the ball park of two days per server.

Service Pack 2

You’re right I didn’t include Service Pack 2. Service Pack 2 is an optional download. You can test this out very easily. Go to any server running Microsoft Hyper-V Server 2008 or Windows Server 2008 and run Windows Update.

Here’s what I included:

Both Microsoft and VMware also have optional patches, but I didn't include this type because they are optional, so we could focus on the most acute patches. Finally, let me very clear about the Microsoft patches. These counts include ALL critical and recommended patches, meaning if there was a critical patch for IE or some other Windows component I counted it. Had I ignored these types of patches, that wouldn’t be a fair comparison.

However, for the sake of comparison and transparency, I added Service Pack 2 to the count and here’s what we found:

• Microsoft Hyper-V Server 2008: 27 patches, totaling 845 MB.
• VMware ESXi 3.5: 13 patches, totaling over 2.7 GB.

To put it another way,

• VMware ESXi 3.5 still had a 4.3x greater patch footprint

Microsoft Hyper-V Server 2008 RTM’d Later Than Windows Server 2008

That’s a great point and my mistake. I apologize for that error and have corrected it. (BTW: It doesn’t take me a month to apologize and make corrections, but I digress..)

Let me tell you how this happened.

In doing the patch comparison, I started with VMware’s favorite example, VMware ESXi 3.5 and Windows Server 2008 which RTM’d in Q4 2007. I wanted to take a reasonable, historical sample size of both platforms. Going back 18 months ensures the sample set covers the majority of Windows Server 2008 time in market and tackles VMware’s fundamental claim that because their disk footprint is smaller and ESX/ESXi are single purpose hypervisors, they are therefore more secure. If anything this should work in VMware’s favor.

As I pored through data I realized I wanted to do the full matrix and include:

1. VMware ESXi
2. VMware ESX
3. Windows Server 2008 (Full)
4. Windows Server 2008 (Core)
5. and finally Microsoft Hyper-V Server 2008

This way, customers get the most comprehensive data and can make an informed decision. As you can see, MS Hyper-V Server 2008 was at the end of the analysis and this mistake crept because it slipped my mind that MS Hyper-V Server 2008 shipped after Windows Server 2008. (For example, MS Hyper-V Server 2008 R2 and Windows Server 2008 R2 are sim shipping this time around…)

The Notion Of Truth

This is an ideal time to make an important and fundamental point about competitive comparisons. The reason we’re able to perform this analysis and debate this topic openly is because the facts are freely available in the public domain without restriction. I’m happy to discuss the methodology used and make corrections as needed, but the point is we can have these discussions.

This point may sound obvious, reasonable and rational, but it’s not what VMware believes when it comes to competitive benchmarking. VMware has decided that they want to tightly control the concept of truth and only want the public to hear the VMware version of the truth. This is an unfortunate situation for customers who value open debate and basic fairness.

That’s another blog for another time.

Cheers, -Jeff

Virtualization Nation,

After my recent blog discussing the release of Microsoft Hyper-V Server 2008 R2, we received overwhelmingly positive feedback. At the same time, there's still some skepticism about free Live Migration and almost daily we keep hearing, "This is too good to be true, is Live Migration really free? Is High Availability also free? What's the catch?" Yes, both Live Migration and HA are free. Check out this earlier blog for details.

At the same time, we've also received questions about Windows Server 2008 Hyper-V, Microsoft Hyper-V Server 2008 and VMware ESX/ESXi in terms of disk footprint. The disk footprint argument is a favorite bit of FUD by VMware which appears to making the rounds again in the blogosphere. In the past couple of weeks, I've seen a few articles reprinting this pabulum almost verbatim. So today, I thought we'd analyze the whole disk footprint argument and, as usual, let's analyze the facts.

The Thin Hypervisor Debate In A Nutshell

Rather than attempt to restate VMware's argument, let's just use their exact verbiage. From VMware's website:

It's interesting to point out that VMware uses ESX & ESXi 3.5 in the column title, but then only uses ESXi for comparison in the right hand column and then rushes to compares ESXi to Windows Server 2008 Hyper-V.

Gosh, I wonder why ESX 3.5 isn't included? We'll cover that too.

Pay special attention to the last paragraph:

"VMware ESXi is a fully functional hypervisor in a 32 MB disk footprint, which reduces the risk of down time and increases reliability."

There are three metrics to VMware's argument:

1. Disk footprint. Disk footprint. Disk footprint. They're quite fixated on that.
2. Reduce the risk of downtime due to fewer patches (i.e. fewer patches is good)
3. Increase reliability (and, in turn, availability)

VMware's overarching point is because their disk footprint is smaller and ESX/ESXi is a single purpose hypervisor, they are therefore more secure. If that's the case, then it stands to reason that ESX/ESXi:

• should have fewer patches (they have less code to patch)
• patches should be smaller in disk footprint (they have a smaller codebase and you want to keep code churn to a minimum; otherwise one could ship a 1k stub file and claim to be smaller)
• should offer higher availability, reliability and uptime

We're going to put that to the test.

Before We Get Started

Before we delve into the analysis, I'd like to point out that VMware touts ESXi as a 32 MB hypervisor, yet the download is over 200 MB. So, are we too assume that the other 170+ MB doesn't count? I point this out because both Windows Server 2008 Hyper-V and Microsoft Hyper-V Server 2008 do have larger disk footprints, but also provide incredible value and capabilities that our customers desire such as driver compatibility with a vast catalog of hardware, widely used management interfaces, scripting capability (PowerShell anyone?), MPIO, High Availability, etc.

If you really want to focus on the disk footprint that matters, the amount of software that could be directly exposed to VM attack, the Hyper-V hypervisor and virtualization stack combined is about 20 MB, ~19.4 MB for the virtualization stack and ~600k for the hypervisor.

In short, VMware has focused on our entire footprint which is made up mostly of stuff that isn't exposed to VM traffic at all or only exposed indirectly, while ignoring the part that matters most and for which VMware doesn't have a strong track record.

Time For Some Analysis

With VMware's own metrics in mind, I decided we should perform an apples to apples comparison. Specifically, over a 12 month period from June 30th 2008 to June 30th 2009, comparing: Microsoft Hyper-V Server 2008 (R1) and VMware ESXi 3.5 in terms of numbers of patches, size of patches and availability. I specifically chose these versions because I wanted to take a reasonable, historical sample size (12 months) of both platforms. Both Hyper-V R2 and VSphere have RTM'd within the last 90 days and that wouldn't be statistically relevant.

What did we compare? Wanting to keep the comparisons as close as possible, here's what we analyzed:

• VMware: Security and Critical Patches
• Microsoft: Critical and Important Patches

Both Microsoft and VMware also have optional patches, but I didn't include this type because they are optional, so we could focus on the most acute patches.. Finally, let me very clear about the Microsoft patches. These counts include ALL critical and recommended patches, meaning if there was a critical patch for IE or some other Windows component I counted it. Had I ignored these types of patches, that wouldn’t be a fair comparison.

Comparison #1: Microsoft Hyper-V Server 2008 & VMware ESXi 3.5

Disk Footprint & Patch Count. Here's what we found:

• Microsoft Hyper-V Server 2008: 26 patches, totaling 82 MB. 
• VMware ESXi 3.5: 13 patches, totaling over 2.7 GB.

Yes, I said over 2.7 GB. To put it another way,

• VMware ESXi 3.5 had a 33x greater patch footprint

So much for the disk footprint argument. How can the ESXi patch footprint be so huge?

Because VMware releases a whole new ESXi image every time they release a patch. Furthermore, because VMware releases a whole new ESXi image every time they release a patch it also means that every ESXi patch requires a reboot. At this point, a VMware salesman, may concede the point that every ESXi server has to be rebooted for every patch, but they will then state that they have VMotion (Live Migration), so it doesn't affect their uptime.

Except when their own patches cause days of downtime and render VMotion impotent.

Reliability/Availability. With VMware ESXi 3.5 Update 2, it included a serious flaw which resulted in two days of downtime for their customers including the loss of VMotion:

"Starting this morning, we could not power on nor VMotion any of our virtual machines," said someone identified as "mattjk" on a VMware support forum. "The VI Client threw the error 'A general system error occurred: Internal Error.'"

It was so bad, VMware's CEO had to apologize on numerous occasions. (HERE, HERE, HERE, HERE, HERE, & HERE). VMware then rushed out the VMware ESXi 3.5 Update 3 which introduced instability to VMware High Availability and could cause virtual machines to spontaneously, unexpectedly reboot. (HERE & HERE)

Virtual machines that unexpectedly reboot due to bugs in VMware high availability. Think about that for a minute.

The News Gets Worse

Not only did VMware ESXi have a 31x greater patch footprint, but they also had the most serious virtualization security flaws. Specifically,

Guest code able to break out of VM and into the ESX Hypervisor

For those you who may not know, code running in a guest operating system that is able to break out of a VM and into a hypervisor is THE WORST type of security flaw you can have for a virtualization platform. Period. VMware has had exploits on both their "thin hypervisor (ESXi)" and on their "thick hypervisor (ESX)" as recently as a few months ago.

For example, April 2009 CVE-2009-1244: A critical vulnerability in the virtual machine display function allows a guest operating system users to execute arbitrary code on the host OS.

Impact: Provides administrator access, Allows complete confidentiality, integrity, and availability violation; Allows unauthorized disclosure of information; Allows disruption of service.

Something to consider when VMware wants to host your data in "their cloud."

Now consider the fact that there were two significant quality and reliability issues with two major updates in a row (ESX/ESXi Update 2 & Update 3). While the initial Microsoft Hyper-V Server 2008 release didn't offer Live Migration (MS Hyper-V Server 2008 R2 now includes Live Migration/HA for free), it didn't include two days of potential downtime and virtual machines unexpectedly rebooting either. For those that track availability in terms of nines (five nines is 5.26 minutes of downtime a year) VMware Update ESXi 3.5 Update 2 dropped customers to "two nines" of availability.

Using VMware's own metrics, Microsoft Hyper-V Server 2008 is clearly the winner over ESXi 3.5.

One more interesting observation

After VMware released Update 2, which affected both ESXi & ESX, they rushed to get a fix out the door and did so in a few days. One interesting thing I noticed was that in the span of a few days, the patch grew 17 MB.

17 MB in a few days????

Now, I'm not sure what's in that 17 MB (potentially more than 50% of their hypervisor footprint could have been churned using VMware's 32 MB hypervisor claim, but I digress.), but if I had just released a patch that resulted in multiple days of downtime for my customers let me tell you what I would have done:

Created a highly scoped, well tested, surgical fix with minimal code churn.

To see a 17 MB change in just a few days would be cause for concern and an immediate code review.

Maybe that's just me.

In my next blog, we'll compare Windows Server 2008 Hyper-V and VMware ESX 3.5.

Cheers,

Jeff Woolsey

Principal Group Program Manager

Windows Server, Hyper-V

Virtualization Nation,

Just a couple of weeks ago, we surprised a few folks by contributing over 20,000 lines of source code to the Linux kernel under GPL V2. Here are some quotes from the announcement:

Our work in this area is all about providing more flexibility and choice, and requests from our customers and partners were really the impetus behind those efforts. We are hearing more and more customers and open source partners telling us they see some of their best value when they deploy new open source software solutions on top of existing Microsoft platforms. Today's release would have been unheard of from Microsoft a few years ago, but it's a prime example that customer demand is a powerful catalyst for change. -Sam Ramji, Sr. Director, Server & Tools, Microsoft

Customers have told us that they would like to standardize on one virtualization platform, and the Linux device drivers will help customers who are running Linux to consolidate their Linux and Windows servers on a single virtualization platform, thereby reducing the complexity of their infrastructure. Consequently, they'll have more choices in how to develop and deploy solutions, while still managing their entire data center from a single management console. -Tom Hanrahan, Director Open Source Technology Center, Microsoft

What Do Analysts Think?

Reviewers and analysts agree this a very big and positive step:

"This is a logical, but precedent-setting decision for Microsoft. Credit Microsoft for recognizing the reality that a sizable portion of its customer base was going to be running Linux and Microsoft side by side in virtualized environments, so it would be important to be competitive on an interoperability front," said Stephen O'Grady, analyst at Redmonk. "For all of its logic, however, this is a move that would have been inconceivable a few years ago, meaning that the glasnost of Microsoft vis a vis open source continues." Stephen O'Grady, InfoWorld

"This is a big deal. When you get in the mainline Linux kernel it is a competitive advantage for Microsoft." Chris Wolf, Network World

What Does Linus Think?

Linus Torvalds provided his own assessment on the Microsoft contribution:

I agree that it's driven by selfish reasons, but that's how all open source code gets written! We all "scratch our own itches". It's why I started Linux, it's why I started git, and it's why I am still involved. It's the reason for everybody to end up in open source, to some degree.

So complaining about the fact that Microsoft picked a selfish area to work on is just silly. Of course they picked an area that helps them. That's the point of open source - the ability to make the code better for your particular needs, whoever the 'your' in question happens to be.

Does anybody complain when hardware companies write drivers for the hardware they produce? No. That would be crazy. Does anybody complain when IBM funds all the POWER development, and works on enterprise features because they sell into the enterprise? No. That would be insane.

So the people who complain about Microsoft writing drivers for their own virtualization model should take a long look in the mirror and ask themselves why they are being so hypocritical. Linus Torvalds

Jason Perlow, Takes A Look At Hyper-V R2 & Few Linux Distros

Jason Perlow, a ZDnet columnist who has been writing about Open Source for over 10 years, was so intrigued by the new capabilities in Windows Server 2008 R2 Hyper-V and Microsoft Hyper-V Server 2008 R2 that he spent the last week installing various Linux distros (CentOS, Scientific Linux, OpenSUSE) atop Hyper-V R2. The result is a detailed article on ZDnet documenting the step-by-step instructions for each distro!

This article makes for an excellent read. I highly recommend checking out Jason's article here at ZDnet.

Cheers,

Jeff Woolsey

Windows Server, Hyper-V

Virtualization Nation,

Just a quick blog in case you missed the big news for Windows 7. Windows XP Mode has hit Release Candidate. Here are some highlights of the new Release Candidate:

New Features in Windows XP Mode RC

Based on feedback from the Windows XP Mode beta, we've made several improvements to the usability of Windows XP Mode for small and medium-sized business users:

• You can now attach USB devices to Windows XP Mode applications directly from the Windows 7 task-bar. This means your USB devices, such as printers and flash drives, are available to applications running in Windows XP Mode, without the need to go into full screen mode.
• You can now access Windows XP Mode applications with a "jump-list". Right click on the Windows XP Mode applications from the Windows 7 task bar to select and open most recently used files.
• You now have the flexibility of customizing where Windows XP Mode differencing disk files are stored.
• You can now disable drive sharing between Windows XP Mode and Windows 7 if you do not need that feature.
• The initial setup now includes a new user tutorial about how to use Windows XP Mode.

Personally, all the apps I use work just fine with Windows 7 (zero app compatibility issues), but I've found one killer reason for Windows XP Mode. At home, we have an HP Photosmart printer that prints great photos. It's really a great printer, but unfortunately, HP never updated the drivers for this printer beyond XP. I haven't had the heart to recycle it just yet and now I don't need to. I setup Windows XP Mode on my system at home, hooked up this USB device to my Windows XP Mode virtual machine, setup printer sharing within XP, and voila we can use this printer again. Very cool. :-)

For more on Windows XP Mode and download location, check out this blog post from the Windows Client Team.

Cheers,

Jeff Woolsey

Principal Group Program Manager

Windows Server, Hyper-V