All benchmarking results should have one of two results:
When we do Hyper-V proof of concepts (POCs) comparisons at a customer site we try to work with the client to setup system configurations where both parties get a fair shake. The client typically sets up their test environment as it’s currently configured in production (and we set up ours), and, just as important, the client runs through a series of tests with relevant workloads on both virtualization platforms.
An important goal of these POCs is to dislodge the perceived status quo, namely that the incumbent leader VMware has a significant performance and feature advantage. Customers who are solid VMware shops are a bit surprised to see there is no performance gap, indeed most of the time Hyper-V performs good as or better than than ESX.
Of course when such results are publicly demonstrated, like in the Virtualization Review Hypervisor comparison published last week, the reaction is a bit more controversial. Let’s say the folks at VMware were not amused that ESX performed so mundanely, and the PR team even had to send out an emergency bulletin to help their field “parse'” the results. VMware has much more to lose if the status quo gets a bit more blurry so their immediate reaction has been to challenge the methodology of the benchmark tests.
I think the authors of the report were very even handed and didn’t just jump to categorical conclusions. The report almost felt like a mea culpa from a ESX fan who painfully realizes that perceived notions of superiority didn’t materialize. I have seen very similar reactions when we perform our POCs and line up our product against the competition.
VMware is the incumbent in server virtualization market and they have managed to cultivate a reputation performance and stability. However, the magic sauce guarded by VMware’s EULA (section 3.3 which prohibits publishing of benchmarks) has shown to be nothing more then some ketchup and mayonnaise. In fact, VMware came last in many of the recently published benchmarks, after Hyper-V and Xenserver, suggesting they are falling behind in the technology game (VMware, for example, does not leverage hardware assisted virtualization and still relies on hardware emulation).
Most of the virtualization blogosphere is now starting to realize that VMware isn’t the only virtualization game in town and more importantly, no longer the safe choice. Due diligence now requires more than just a cursory look at competitor datasheets – and benchmarking is that next step. And the more test drives customers do with Hyper-V the more they are driving away with our technology.