Hypervisor Footprint Debate Part 3: Windows Server 2008 Hyper-V & VMware ESXi 3.5

Hypervisor Footprint Debate Part 3: Windows Server 2008 Hyper-V & VMware ESXi 3.5

  • Comments 21
  • Likes

In my last two blog posts (Part 1 & Part 2), I started an in depth analysis tackling VMware's claims head on that because their disk footprint is smaller and ESX/ESXi are single purpose hypervisors, they are therefore more secure. If that's the case, then it stands to reason that ESX/ESXi:

  • should have fewer patches (they have less code to patch)
  • patches should be smaller in disk footprint (they have a smaller codebase and you want to keep code churn to a minimum; otherwise one could ship a 1k stub file and claim to be smaller)
  • should offer higher availability, reliability and uptime

Using VMware's own metrics:

In part 3, lets take a look at VMware's favorite comparison Windows Server 2008 Hyper-V to ESXi 3.5. Let's have a look.

Stacking The Deck In VMware's Favor

In this last comparison, I will freely admit that this isn't an apples to apples comparison. In this comparison, I gave ESXi 3.5 a 6 month advantage. Here's what I mean. Specifically, I compare:

  • VMware ESXi 3.5 from June 30th 2008 to June 30th 2009 (a 12 month period)
  • Windows Server 2008 Hyper-V January 1 2008 to June 30th 2009 (an 18 month period)

Using an 18 month sample set for Windows Server 2008 covers the majority of its time in market and goes to the heart VMware's fundamental claim that because their disk footprint is smaller and ESX/ESXi are single purpose hypervisors, they are therefore more secure. This tilts the scale in VMware's favor.

Windows Server 2008 Hyper-V to VMware ESXi 3.5

Disk Footprint & Patch Count. Here's what we found:

  • Windows Server 2008 Full Installation: 32 patches totaling 408 MB of patches
  • Windows Server 2008 Core Installation: 26 patches totaling 82 MB of patches or (~20% fewer than a Windows Server 2008 full installation)
  • VMware ESXi 3.5: 13 patches, totaling over 2.7 GB.

Yes, I said over 2.7 GB. To put it another way,

  • VMware ESXi 3.5 had a 6.6x greater patch footprint than Windows Server 2008 (Full)
  • VMware ESXi 3.5 had a 33x greater patch footprint than Windows Server 2008 (Core)


So much for the disk footprint argument. Again, how can the ESXi footprint be so huge?

Because VMware releases a whole new ESXi image every time they release a patch. Furthermore, because VMware releases a whole new ESXi image every time they release a patch it also means that every ESXi 3.5 server requires a reboot. At this point, a VMware salesman may actually concede that every ESXi server has to be rebooted for every patch, but they will then state that they have VMotion (Live Migration) so it doesn't affect their uptime.

Except when their own patches cause days of downtime and render VMotion impotent.

Reliability/Availability. With VMware ESXi 3.5 Update 2, it included a serious flaw which resulted in two days of downtime for their customers including the loss of VMotion:

"Starting this morning, we could not power on nor VMotion any of our virtual machines," said someone identified as "mattjk" on a VMware support forum. "The VI Client threw the error 'A general system error occurred: Internal Error.'"

It was so bad, VMware's CEO had to apologize on numerous occasions. (HERE, HERE, HERE, HERE, HERE, & HERE). VMware then rushed out the VMware ESXi 3.5 Update 3 which introduced instability to VMware High Availability and could cause virtual machines to spontaneously reboot. (HERE & HERE)

Virtual machines that spontaneously reboot due to bugs in VMware high availability.

Now consider the fact that there were two significant quality and reliability issues with two major updates in a row (ESX/ESXi Update 2 & Update 3). While the initial Windows Server 2008 Hyper-V release didn't provide Live Migration (Windows Server 2008 Hyper-V R1 had Quick Migration and Windows Server 2008 Hyper-V R2 includes Live Migration for free), it didn't include two days of potential downtime and virtual machines unexpectedly rebooting either. For those that track availability in terms of nines (five nines is 5.26 minutes of downtime a year) VMware Update 3.5 Update 2 dropped customers to "two nines" of availability.

Using VMware's own metrics, Windows Server 2008 Hyper-V is clearly the winner over ESXi 3.5.

The Facts Contradict VMware's Claims

As stated at the beginning of this series, VMware's overarching point is because their disk footprint is smaller and ESX/ESXi is a single purpose hypervisor, they are therefore more secure. While VMware heavily touts this claim (it's in numerous location on their website for starters), the facts from this analysis directly contradict their claims. Specifically:

  1. The platform with the largest number of patches was VMware ESX 3.5 with 85 Security & Critical Patches averaging over a patch per week.
  2. The platform with the largest patch footprint was VMware ESX 3.5 totaling over 3 GB worth of patches followed by VMware ESXi 3.5 with over 2.7 GB. That's right, VMware's single purpose virtualization platforms that claims to have the smallest footprint had the two largest patch footprints by about a mile. (Graph below)
  3. Both VMware ESX & ESXi had a recent case of the most severe virtualization flaw with guest code able to break out of the virtual machine and could potentially:
    • Provides administrator access, Allows complete confidentiality, integrity, and availability violation; Allows unauthorized disclosure of information; Allows disruption of service.
  4. VMotion/Live Migration is not a panacea to patching. It can help, but in the case of VMware's own self-inflicted faulty patch, it rendered their advantage impotent.
  5. VMware had not just one, but two significant updates with serious quality and reliability issues with both ESX and ESXi. Specifically, ESX/ESXi Update 2 Issues: HERE, HERE, HERE, HERE, HERE, & HERE & ESX/ESXi Update 3 Issues: HERE & HERE.


The Point Of This Series

Say it with me:

>  Security is more than just disk footprint. <

Quoting disk footprint size alone is a nice pithy, superficial phrase, but it's also a boat load of bollocks. The next time some VMware representative throws out that argument, point them to this blog and tell them Jeff sent you. If you've ever spent anytime with a security expert, one of the first things they will tell you is that security is not a one time exercise. Security is an ongoing process that should be embedded throughout the entire development lifecycle. It's that belief that drove us to develop the Microsoft Secure Development Lifecycle (SDL) and is publicly available.

Microsoft Secure Development Lifecycle (SDL)

The concepts that make up the Microsoft SDL were formed with the Trustworthy Computing (TwC) directive of January 2002. At that time, many software development groups at Microsoft instigated "security pushes" to find ways to improve the security of existing code.

Becoming a mandatory policy in 2004, the SDL represents a major cultural evolution at Microsoft with regards to software security and privacy and has matured into a well defined methodology. A "security process by a software company," the SDL was designed as an integral part of the development process. The development, implementation and constant improvement of the SDL represents a strategic investment for Microsoft, and an evolution in the way that software is designed, developed, and tested.

From a high level, the Microsoft SDL looks like this:

The Microsoft Security Development Lifecycle

Benefits of the Microsoft SDL:

  • Reducing the number of software vulnerabilities

    The SDL has played a critical role in embedding security and privacy into Microsoft software and culture, leading to measurable and widely recognized security improvements in flagship products such as Windows and SQL Server and the proof is real. How about:

    • Windows Server 2008 Full vs Server Core
      • Reduction in patches by ~50%
  • Reducing the total cost of development

    The SDL reduces the "total cost of development" by finding and eliminating vulnerabilities early. According to the National Institute of Standards and Technology (NIST), eliminating vulnerabilities in the design stage can cost 30 times less than fixing them post release.

Read that last sentence again.

Thirty times.

I also want to point out that the Microsoft Security Development Lifecycle doesn't end once the bits are released. It also means having a well-established response mechanism including:

  • responding to potential security threats
  • root cause analysis to understand why the issue occurred and ensure that issue isn't repeated
  • issuing security patches

The importance of a security development lifecycle cannot be understated. No matter how well you execute, there is no such thing as perfect code. Whether it's Microsoft, VMware, <insert software vendor here>, having a rigorous security development practices in place is imperative. And, in case you think I'm satisfied with our patch numbers above, you'd be wrong. I don't ever want to get complacent and think for a moment that "security is done." Security is never done.

Let he who has written perfect code throw the first stone.

Let's Have A Look At The VMware's Security Development Lifecycle

So, where's the VMware Security Development Lifecycle?

You're guess is as good as mine.

I went to VMware's site and searched for their security development lifecycle. I found their Security Center which lists their patches, but that's just one small aspect to a security development lifecycle. I Bing searched "VMware security development lifecycle" and was returned a sales pitch from VMware to buy something at $1500 per processor.

No, I'm not kidding.



Making The SDL Available To Our Partners

After a significant investment in time, money, manpower we've developed and want to give back to our partners. A great place to start is the Microsoft SDL Homepage. Here you will find whitepapers, best practices, threat modeling tools, process guidance and much more. In addition, we recently released the Microsoft SDL Process Template for Visual Studio Team Systems. This template helps ease the adoption of the SDL, demonstrates security return on investment and provides auditable security requirements and status.

I'd be remiss if I didn't point out an excellent book aptly titled, Writing Secure Code Vol. 2 and point to the blog of one of the authors, Michael Howard. More links below.

Microsoft SDL Homepage: http://msdn.microsoft.com/en-us/security/cc448177.aspx

Microsoft SDL Process Template for Visual Studio Team System: http://msdn.microsoft.com/en-us/security/dd670265.aspx

Writing Secure Code Volume 2: http://www.amazon.com/Writing-Secure-Second-Michael-Howard/dp/0735617228

Michael Howard's Blog: http://blogs.msdn.com/michael_howard/

In my next blog, we'll discuss more free tools and programs available our partners.


Jeff Woolsey

Principal Group Program Manager

Windows Server, Hyper-V

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • You will tell that lie that ESXi 3.5 has more patch footprint over and over again, uh ? Tell a lie 1000 times, and it becomes a true.

    The disk footprint os ESXi 3.5 is NOT huge as you mention, you are manipulation the data (as all patches replaces the whole image).

    I will tell it again something you are being dishonest to admit:

    Hyper-V on Server core has TWICE more security problems as ESXi 3.5, according to your own analysis.

    Now you can do part IV and say it all over again.

    By the way, this update2 problem is ancient history. How much time more you will remember that ? Should we start remembering Windows horrible security record, exploits, and all this old stuff ?

  • Unfortunatly I think you have shot yourself in the foot by repeating the same informtation over the 3 parts, and harped on and on about the vmotion bug (yes it was/is a big issue, but repeating the same thing over 3 posts makes you look petty)

    Maybe you should rewrite the 3 posts as 1, do some serious editing, remove all the same info that is cut/pasted.  

    In short, part 1 was good, part 2 and 3 jumped the shark very fast

  • This is getting to be boring with all the bashing etc. Get onto the decent stuff already. People got the point in part 1 already. NEXT.

    To prove your points that Hyper V is a better product to sway the "customer", get some definitive data up here, give us the best practises and all the good stuff Microsoft have internally.

  • Let's start with this quote:

    "VMware's claims head on that because their disk footprint is smaller and ESX/ESXi are single purpose hypervisors, they are therefore more secure"

    This doesn't make any sense, footprint isn't related to security in any way. But still you want to spend a few hunderd words on a worhtless claim from VMware.

    Then you explain why VMware actually has such a big footprint: they release a complete new image every time. So the patches don't have to be bigger or smaller then Hyper-V, you are comparing the complete VMware image against just the patches of Hyper-V. So, as you can understand that is just a rubbish comparison. You even agree with this later on: "(...)but it's also a boat load of bollocks".

    But the main point I want to make here, just stop bashing each other, and try to improve your products. Sure VMware has made a mistake with that Update 2, and as we all know Hyper-V isn't what you call 'cutting edge'. But your product isn't getting better by exposing the flaws of your competetor.

    So stop wasting time and go and improve Hyper-V. It can use a lot of that.  Just try to get better than your competetors instead of trying to make them look bad.

  • Have now removed this blog from my RSS feeds.  Everytime I read an article it just focuses on bashing the competition.  

    I thought the role of a principal program manager  was to promote their product?  

  • Being both a Microsoft customer and a VMware customer, I find that this kind of blogging only puts the blogger and the bloggers company in a undefendable position. It only shows an unbelievable lack of seriousness and will push more customers away than it will attract and I'm quite frankly offended by such postings and our usual approach is to make it clear to all our Microsoft employee contacts that this does not put Microsoft in any good spotlight at all at our business. Could we please see some serious blogs about Hyper-V qualities and why I as a customer should be running Hyper-V instead of a competing product.

  • In opposition, you will not find VMware bashing the competition on this non-professional way.

    You do not see those angry blog posts from VMware anywhere, only serious and professional product comparing.

    Amateur stuff like the 10-myths video does not help MS reputation.

  • Wow - are we really that bothered about size of patches? I have 750GB of hard disk on my laptop - enough for 250 years worth of ESX patches. I'm presuming Hyper-V will have as rich a feature set as the VMWare product range currently commands by then.

  • AC666, only serious and professional product comparing from VMware? Are you kidding me? These blog posts here that you are complaining about are in response to VMware FUD. How about the bogus crash video put out by VMware. I'm sorry but VMware is putting out false information left and right, so I don't take issue with the MS guys looking to set some things straight. The whole disk footprint thing is something the VMware sales guys beat like a dead horse, so I'm fine with some response from MS here. I think the discussion of the SDL is a key thing.

  • optimus,

    Nothing compares to this kind of FUD, like the 10-myths video, and other misleading information MS is spreading (ESX is an additional layer, but Hyper-V is not ? for ex).

    Here we clearly see an intentional error on patch footprint: The guys is making the wrong calculations to hide the truth, which is, Hyper-V on server core has twice more security problems than ESXi, and much higher footprint.

    Make some defense is legitimate, but with real facts, and not misleading information.

    The Hyper-V crash video was a mistake, and initiative from a VMware employee , which apologized latter, acknowledging the mistake.

    Here we have a MS Program Manager using his time to make some terribly flawed analysis.

    The comments here says it all about people's perception.

  • AC666, I'm not seeing where you see a lie, or "wrong calculations to hide the truth" VMware releases the whole kit and kaboodle over agin each time, so yes it has a larger patch footprint in MB. That isn't a false statement. Now if you want to talk number of patches instead because you think you have the advantage there, let's notice your own and VMwares comparison is the one that is flawed. You want to compare ESXi to full Hyper-V. That isn't a fair comparison. If you want to compare it needs to be Hyper-V vs. ESX. Which is 26 patches for Hyper-V and 85 for ESX. Which by your logic would make ESX 3 times more insecure than Hyper-V. However, like Jeff is pointing out security is more than just footprint or number of patches. First he blows up VMwares lies, then gets into what really means security. Things like the SDL, VMwares lack of an equivalent SDL, and the recent significant vulnarbilities in ESX.

  • ESXi --> 13 patches

    W2K8 server core --> 26 patches

    Normally 1 patch corrects 1 security problem.

    Twice more vulnerabilities. The patch size is completely *irrelevant*. ESXi is still ~200 MB, no matter how much patches you apply to it.

    Of course it is a fair comparison. A security concerned customer might choose between the 2 offerings. It only happens MS does not have a 200MB hypervisor.

    Of course security is not only abut the number of patches, but that is what Jeff is stressing here.

  • Let's be fair here - this article is a great example of the 'devil in the detail' - one can pick a specific issue (in this case patching), and spin the figures to come out one way, or another - and of course, Microsoft has spun them in their favour.

    Step back and take a look at the bigger picture however, and no-one in their right mind would say that overall, ESXi is a 'bigger' product than Windows 2008 - and yes, I am fully aware of server core in all this.

    The simple fact of the matter is that ESX and ESXi are both more streamlined hypervisors than Windows 2008 - but feel free to keep on spinning the figures.

  • AC666, you still don't seem to be catching my point, so let me break it down.

    1. ESXi does not equal W2k8 w/ Hyper-V. A better comparison is ESX to W2k8 w/ Hyper-V. This is the comparison that VMware in fact talks about in their marketing material EXCEPT when it comes to doing the numbers they go back to using ESXi because it had 13 patches while ESX had 85.

    2. Number of patches is no better a measure of security than patch size. Jeff isn't even the one claiming path size or count matter for security. That is the VMware claim. That their smaller disk footprint makes them more secure. Is a house with a crumblign foundation better than a house with a burnt out light bulb and a leaky faucet because the house with the bad foundation only has one problem not two? No. You have to look at severity of the issue. VMwares vulnerability of being able to break out into the host is the biggest vulnerability a hypervisor can have.

    3. That is NOT what Jeff is stressing here. Did you read the post? That is what VMWARE is stressing and Jeff is disspelling. First he points out that even by their own flawed metric of security they aren't really champs. Then he talks about what really means security and that is having a secure development process to prevent vulnerabilities from getting into the code and a process for dealing with fixing those vulnerabilities when they do make it in. Vmware shows no evidence of having either as their updates a 'fixes' tend to make even bigger breaks.

  • Optimus

    1) Forget about comparing. VMware has a 200MB hypervisor, and MS don't. A customer has both choices, so, why not compare if you are evaluating ? There's no correspondence between ESX Full vs W2K8 , or ESXi vs W2K8 Server core, or whatever.

    Also, size really matters. Compare ESX with ESXi. ESXi has much less security patches than the full ESX, which proves that removing the general propose OS is good.

    MS did the server core version of Windows, which has the same benefits: Less stuff, less security problems.

    2) Of course the number of patches if a better indication than the size of patches itself. No matter if the patch has 1KB or 1GB, it corrects a security problem. If a product has 50 vulnerabilities, and another one has 5, what exposes more to a external break ? What changes if a patch has 10 or 50 MB ? Also, the sum of patch sizes Jeff did is deeply flawed !

    Of course we should take into account how severe is the vulnerability, you are very right on this one, and in that point, Windows has a bad security record.

    3) What is VMware is stressing is the Hypervisor footprint. No matter how much you patch ESXi, it is still around 200MB. Taking away unneeded components is proven to have much less exposure to problems and security issues. MS DOES THE SAME WITH SERVE CORE, taking away unneeded components. It just happens it still is a general propose OS, and not a thin 200MB hypervisor built solely for that propose.