Hypervisor Footprint Debate Part 1: Microsoft Hyper-V Server 2008 & VMware ESXi 3.5

Hypervisor Footprint Debate Part 1: Microsoft Hyper-V Server 2008 & VMware ESXi 3.5

  • Comments 11
  • Likes

Virtualization Nation,

After my recent blog discussing the release of Microsoft Hyper-V Server 2008 R2, we received overwhelmingly positive feedback. At the same time, there's still some skepticism about free Live Migration and almost daily we keep hearing, "This is too good to be true, is Live Migration really free? Is High Availability also free? What's the catch?" Yes, both Live Migration and HA are free. Check out this earlier blog for details.

At the same time, we've also received questions about Windows Server 2008 Hyper-V, Microsoft Hyper-V Server 2008 and VMware ESX/ESXi in terms of disk footprint. The disk footprint argument is a favorite bit of FUD by VMware which appears to making the rounds again in the blogosphere. In the past couple of weeks, I've seen a few articles reprinting this pabulum almost verbatim. So today, I thought we'd analyze the whole disk footprint argument and, as usual, let's analyze the facts.

The Thin Hypervisor Debate In A Nutshell

Rather than attempt to restate VMware's argument, let's just use their exact verbiage. From VMware's website:


It's interesting to point out that VMware uses ESX & ESXi 3.5 in the column title, but then only uses ESXi for comparison in the right hand column and then rushes to compares ESXi to Windows Server 2008 Hyper-V.

Gosh, I wonder why ESX 3.5 isn't included? We'll cover that too.

Pay special attention to the last paragraph:

"VMware ESXi is a fully functional hypervisor in a 32 MB disk footprint, which reduces the risk of down time and increases reliability."

There are three metrics to VMware's argument:

  1. Disk footprint. Disk footprint. Disk footprint. They're quite fixated on that.
  2. Reduce the risk of downtime due to fewer patches (i.e. fewer patches is good)
  3. Increase reliability (and, in turn, availability)

VMware's overarching point is because their disk footprint is smaller and ESX/ESXi is a single purpose hypervisor, they are therefore more secure. If that's the case, then it stands to reason that ESX/ESXi:

  • should have fewer patches (they have less code to patch)
  • patches should be smaller in disk footprint (they have a smaller codebase and you want to keep code churn to a minimum; otherwise one could ship a 1k stub file and claim to be smaller)
  • should offer higher availability, reliability and uptime

We're going to put that to the test.

Before We Get Started

Before we delve into the analysis, I'd like to point out that VMware touts ESXi as a 32 MB hypervisor, yet the download is over 200 MB. So, are we too assume that the other 170+ MB doesn't count? I point this out because both Windows Server 2008 Hyper-V and Microsoft Hyper-V Server 2008 do have larger disk footprints, but also provide incredible value and capabilities that our customers desire such as driver compatibility with a vast catalog of hardware, widely used management interfaces, scripting capability (PowerShell anyone?), MPIO, High Availability, etc.

If you really want to focus on the disk footprint that matters, the amount of software that could be directly exposed to VM attack, the Hyper-V hypervisor and virtualization stack combined is about 20 MB, ~19.4 MB for the virtualization stack and ~600k for the hypervisor.

In short, VMware has focused on our entire footprint which is made up mostly of stuff that isn't exposed to VM traffic at all or only exposed indirectly, while ignoring the part that matters most and for which VMware doesn't have a strong track record.

Time For Some Analysis

With VMware's own metrics in mind, I decided we should perform an apples to apples comparison. Specifically, over a 12 month period from June 30th 2008 to June 30th 2009, comparing: Microsoft Hyper-V Server 2008 (R1) and VMware ESXi 3.5 in terms of numbers of patches, size of patches and availability. I specifically chose these versions because I wanted to take a reasonable, historical sample size (12 months) of both platforms. Both Hyper-V R2 and VSphere have RTM'd within the last 90 days and that wouldn't be statistically relevant.

What did we compare? Wanting to keep the comparisons as close as possible, here's what we analyzed:

  • VMware: Security and Critical Patches
  • Microsoft: Critical and Important Patches

Both Microsoft and VMware also have optional patches, but I didn't include this type because they are optional, so we could focus on the most acute patches.. Finally, let me very clear about the Microsoft patches. These counts include ALL critical and recommended patches, meaning if there was a critical patch for IE or some other Windows component I counted it. Had I ignored these types of patches, that wouldn’t be a fair comparison.

Comparison #1: Microsoft Hyper-V Server 2008 & VMware ESXi 3.5

Disk Footprint & Patch Count. Here's what we found:

  • Microsoft Hyper-V Server 2008: 26 patches, totaling 82 MB
  • VMware ESXi 3.5: 13 patches, totaling over 2.7 GB.

Yes, I said over 2.7 GB. To put it another way,

  • VMware ESXi 3.5 had a 33x greater patch footprint


So much for the disk footprint argument. How can the ESXi patch footprint be so huge?

Because VMware releases a whole new ESXi image every time they release a patch. Furthermore, because VMware releases a whole new ESXi image every time they release a patch it also means that every ESXi patch requires a reboot. At this point, a VMware salesman, may concede the point that every ESXi server has to be rebooted for every patch, but they will then state that they have VMotion (Live Migration), so it doesn't affect their uptime.

Except when their own patches cause days of downtime and render VMotion impotent.

Reliability/Availability. With VMware ESXi 3.5 Update 2, it included a serious flaw which resulted in two days of downtime for their customers including the loss of VMotion:

"Starting this morning, we could not power on nor VMotion any of our virtual machines," said someone identified as "mattjk" on a VMware support forum. "The VI Client threw the error 'A general system error occurred: Internal Error.'"

It was so bad, VMware's CEO had to apologize on numerous occasions. (HERE, HERE, HERE, HERE, HERE, & HERE). VMware then rushed out the VMware ESXi 3.5 Update 3 which introduced instability to VMware High Availability and could cause virtual machines to spontaneously, unexpectedly reboot. (HERE & HERE)

Virtual machines that unexpectedly reboot due to bugs in VMware high availability. Think about that for a minute.

The News Gets Worse

Not only did VMware ESXi have a 31x greater patch footprint, but they also had the most serious virtualization security flaws. Specifically,

Guest code able to break out of VM and into the ESX Hypervisor

For those you who may not know, code running in a guest operating system that is able to break out of a VM and into a hypervisor is THE WORST type of security flaw you can have for a virtualization platform. Period. VMware has had exploits on both their "thin hypervisor (ESXi)" and on their "thick hypervisor (ESX)" as recently as a few months ago.

For example, April 2009 CVE-2009-1244: A critical vulnerability in the virtual machine display function allows a guest operating system users to execute arbitrary code on the host OS.

Impact: Provides administrator access, Allows complete confidentiality, integrity, and availability violation; Allows unauthorized disclosure of information; Allows disruption of service.

Something to consider when VMware wants to host your data in "their cloud."

Now consider the fact that there were two significant quality and reliability issues with two major updates in a row (ESX/ESXi Update 2 & Update 3). While the initial Microsoft Hyper-V Server 2008 release didn't offer Live Migration (MS Hyper-V Server 2008 R2 now includes Live Migration/HA for free), it didn't include two days of potential downtime and virtual machines unexpectedly rebooting either. For those that track availability in terms of nines (five nines is 5.26 minutes of downtime a year) VMware Update ESXi 3.5 Update 2 dropped customers to "two nines" of availability.

Using VMware's own metrics, Microsoft Hyper-V Server 2008 is clearly the winner over ESXi 3.5.

One more interesting observation

After VMware released Update 2, which affected both ESXi & ESX, they rushed to get a fix out the door and did so in a few days. One interesting thing I noticed was that in the span of a few days, the patch grew 17 MB.

17 MB in a few days????

Now, I'm not sure what's in that 17 MB (potentially more than 50% of their hypervisor footprint could have been churned using VMware's 32 MB hypervisor claim, but I digress.), but if I had just released a patch that resulted in multiple days of downtime for my customers let me tell you what I would have done:

Created a highly scoped, well tested, surgical fix with minimal code churn.

To see a 17 MB change in just a few days would be cause for concern and an immediate code review.

Maybe that's just me.


In my next blog, we'll compare Windows Server 2008 Hyper-V and VMware ESX 3.5.


Jeff Woolsey

Principal Group Program Manager

Windows Server, Hyper-V

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • Lets be honest , while patching Hyper-V you need to apply patches to ALL operating system parts including notepad , IE , outlook express and more!

    in other hand patching vmware its only components that supposed to be in hypervisor , i dont know where you took 3.7 GB , obviously you don't have any experience with vmware . When ESX 3.5  was on market Microsoft did not had any visualization product , and i don't understand why you don't compare current ESX 4.0 VSphere ?

  • Wow... very interesting facts and perspective, Jeff. I am all anxious to see VMware response.

    @semkras, you definitely want to run your Hyper-V on Server Core installation, in particularly to avoid patching of "ALL operating system parts including notepad , IE , outlook express and more"; but more importantly, to reduce attack surface. With SCONFIG out there in R2, there's simply no good reasons or excuses NOT to run Hyper-V on Server Core installation, period.

  • @semkras

    while we're being honest, I have used ESX3-3.5 in live production for the past 2 years and have a 7 node enterprise cluster using FC SAN. I was a member of the HyperV TAP program during its development and have had a 6 node cluster using the same FC SAN in live production since pre RTM.

    With this in mind, I like to think I have a good understanding of both. Sure there is always someone that knows more, but I'm happy I know enough.

    To address your points:

    1) Required to patch IE, OE/WLM, etc?

    If you are installing HyperV in a production environment it is almost certain you are installing the ServerCore SKU (or even HyperV2008). Assuming this is the case patching is minimal and things like OE/WLM do not even exist.

    2) Saying the HyperV program manager has no experience with VMWare - their primary competitor?

    Appologies but perhaps that is a little naive :(

    3) ESX 3.5 was on the market MS did not have any virtualisation product?

    You mean apart from HyperV RTM? Without checking dates I recall ESX3.5 was late Q4 of 2007 and HyperV RTM was late Q1 of 2008 - both have been available since.... so MS did have a product when 3.5 was on the market.

    4) Why not compare current 4.0 VSphere? From my understanding the article is a response to VMWare's own site stating comparison between ESX3.5/ESXi and HyperV RTM.... sure you could compare two different products (4.0 and HyperVR2) but they are not the products VMWare are talking about and not what this article is responding to.

    Don't get me wrong, I like ESX and use it daily. But I also like HyperV and think VMWare are being a little bit "unfair" with some of their recent marketing.

    From a personal opinion - I trust HyperV more than I trust ESX. I have had a couple of trashed VMs on ESX over the last couple of years and was also affected by the Update2 bug - which was "interesting".

    With contrast, my HyperV cluster has just sat there and worked flawlessly since it was installed. This is exactly why my live production VMs (ie - Exchange, DCs, etc, etc) are all on HyperV and I have no plans to change it.

  • Thanks for your comments. I agree with your comment, "Lets be honest , while patching Hyper-V you need to apply patches to ALL operating system parts including notepad , IE , outlook express and more!"

    The counts above INCLUDE ALL THOSE PATCHES. For example, if there was a critical/recommended patch for IE, I included it. Otherwise this wouldn't be a fair comparison. And, it's about time we had a fair comparison.

    Again, thanks for your comments, I made a slight edit in the blog making sure I was clear on that point. Cheers, -Jeff

  • I love these debates, it cracks me up. The best part is, there are the FACTS and not the nonsense and marketing junk you will see everywhere else.

    Keep it up!

  • I was at VMWorld 2007, and I got one of these ESXi copies on a memory stick, a 1GB memory stick, I decided I wanted to make a copy. So I get selfimage and I try to copy it onto a 512MB memory stick, no chance, if you include all the drivers and the hypervisor, it is bigger than 1/2 a Gig - once you have "gunzipped" all the device drivers. So sure the hypervisor is small, but then it's useless without the drivers.

    Now consider the kernel of good old Windows XP - guess how big it is? 70-80 MB. How do I know this, I have been using Citrix Provisioning Server to create boot Windows XP as a vDisk (virtual disk) and the total network requirement to boot the kernel is around 75 MB. Now that’s a five year old graphically rich user centric Operating System. Suddenly VMware’s boasts about its hypervisor don’t seem that impressive!

  • This is something of a spin.

    We're still comparing a streamlined OS (ESX / ESXi) designed to do one thing (virtualize), to a more general purpose OS (Windows).  Sure, there's server core, and Hyper-V Server - but that still doesn't change the fact that the underlying OS, and all of it's supporting components are of a more general purpose nature.

    The footprint is typically larger, and the level of optimization reduced compared to ESX / ESXi

  • @Joe Gough

    2008 Server Core is not a realy a core, its far away from getting even closer to linux core, without KDE/Gnome . All you have its same system without explorer.exe .So pathing is neccesarry for a lot features you dont really need.

    You cant call Virtual Server 2005 virtualization platform for production server , its close to Vmware Workstation product , for developers and test enviroments .

    Anyway Microsoft is going in right direction and if we will having this conversation in 5 years things will be definatly better for Microsoft , but for now its too early to compare HyperV with ESX .

  • @Semkras

    Don't you mean Virtual PC which is close to VMWare Workstation?

    Virtual Server is a virtualisation product in line with VMWare server and before VMWare GSX.

    As for Virtual Server 2005 in a production environment... Why not. If it fits use it. I know that several companies which I worked at used Virtual Server in a production environment.

    I also used VMWare, but I am more a fan of Hyper-V. The one thing that pleases me the most in hyper-v is that when preparing an image for OS deployement in a virtual machine, for Hyper-V I don't have to install any drivers to have it function. So no drivers which I have to delete or maintain on my clients when deployed. This worked instantly. In VMWare ESXi 3.5 it took me nearly 3 days to get this working, and I had to create a custom script which removes all drivers at deploy time.

    But this are just my two cents.


    Love the article.

  • I agree that it's wrong for VMware to lump ESXi with ESX in their marketing materials about disk footprint, since it is less applicable to ESX. In any case, the main patch-related benefit of ESXi disk footprint is not about having reduced filesize of patches, rather, it is about the amount of software that actually needs to be patched.

    It can be argued that disk footprint is not a good metric for this, and the correlation is not perfect, but the proof is above: 13 patches for ESXi versus 26 for Microsoft Hyper-V Server 2008! That's 100% more! Sure, those 13 patches have a bigger filesize, but is it even relevant?

    Another important consideration is that even with the small footprint, ESXi is still manageable by Virtual Center, while Microsoft Hyper-V Server 2008 wasn't manageable by SCVMM until the R2 release, severely limiting it's usefulness in most enterprises (this basically relegates it to a test-only environment). It's great that R2 allows management by SCVMM, since it will be more a viable product.

  • enter in this has been helpful to people constantly thank everyone on behalf of