James O'Neill's blog

Windows Platform, Virtualization and PowerShell with a little Photography for good measure.

Scaling Hyper-v.

Scaling Hyper-v.

  • Comments 14
  • Likes

A couple of Stories have been doing the rounds on our internal Virtualization discussions. One was headed "HOLY COW HYPER-V VIRTUALIZING MICROSOFT.COM!!!!!!" (and before anyone wonders if this is breaking something internal to the world, it's already been described in detail on by Rob Emanuel on the Windows Server blog ). The MS.COM operations team have also produced an article on how they virtualized Technet and MSDN

Now... Microsoft.com is not your average home page. The statistics are staggering: 1.2 Billion hits per month, on average that's over 4,000 every second, but at busy times it peaks at 4 times that. Its 7 million pages take up 300GB of space. It used to need 80 servers to deliver it, but the Ops team migrated it to Server 2008 and newer hardware and saw the opportunity to reduce the number of servers. What became apparent with modern CPUs and RAM sizes, the servers were disk bound Simply throwing more CPU cores and more RAM at the servers wasn't going to reduce the number of boxes needed. Redesigning the site so it used more disk spindles would help - but the quickest win would be to take bigger servers and use Hyper-V to partition them, with each Virtual Machine getting it's own disks that would double the the number of disk IOs available without breaking the site into parts on different disks. But could they use virtualization with out Hyper-V itself becoming the bottleneck. Deploying new servers into the array is involves sync'ing 7 million pages: would virtualizing the servers - even if they ran one VM per box - help deployment ? Even if Hyper-V could scale and wasn't a drag on management and deployment , would it be reliable ?  And would running one Mega site on Hyper-V give the Microsoft.Com folks confidence to consolidate some of the smaller machines they mange. ... incidentally Blogs.technet.com where this page is hosted is run for us by a third party.

If you read the post you find the detail behind why the answers to all these questions turns out to be yes.

I think (at least in the short term) most of the deployments of Hyper-V will be consolidating 5-20 servers into a single box.  It's perfectly capable of running many more VMs than that - indeed we demonstrated hundreds of VMs on the old Virtual Server product - (more than VMware will support) but my own view is that the typical VM requirement, and the typical hardware capacity leads to a typical ratio of 10:1 (it could be 8, or 12 but I'm using rough orders of magnitude here) and the greatest most deployments will fall within half and double that. That's not scientific, but that's how I get to my own "gut-feel". I say servers, because I'm not a great believer in virtualizing the desktop OS - a thin client with a fatter server running your desktop as a VM doesn't reduce hardware costs compared with, rich client and skinnier server architecture. It doesn't reduce power consumption (in fact it probably increases power and A/C costs) and delivers an inferior service; don't try video conferencing company events, or deploying a Voice technology from the the PC. Don't try working off line either. Yet it has the management and licensing overhead of having many machines.  If there isn't really a requirement for a PC , just one or two PC applications, then a terminal Service way of working is usually better.

So, I see Hyper-V most running servers and this case of running a single workload on under Hyper-V - even running multiple identical instances of the same workload is unusual. But if anyone tries to tell you Hyper-V doesn't scale to take on the biggest workloads... well you know different.

In the same vein, QLogic announced Hyper-V can do 180,000 IOPs. That's not a typo. It's vast number of I/O operations per second. In fact some people find it unbelievable, Chris Wolf posted a critique of the test on his blog , the comments are interesting and I felt the need to join in. Chris actually sent me a nice mail afterwards, so I'll repost what I said on in my comment.

The purpose of this benchmark is to prove - if it can be proved - that Hyper-V is not an I/O bottle neck. I read the numbers and said "What the hell kind of system can do 200,000 IOPs per second" it was plainly not the kind of system which is going be installed in many environments. It allows Microsoft people to shout "B.S." at the top of their lungs if anyone from VMware claims to have drivers which are much better than Windows ones. It also kills any suggestion that Hyper-v and Windows drivers are OK in small systems but don't scale.

You're right that if a Microsoft benchmark says "runs at 90% of the speed of RAW hardware" the intelligent question to ask is "is that better, worse or about the same as the competition". Is it "Faster than any previous benchmark on virtulization" because it got a better percentage of the hardware or because it kept on scaling when the hardware improved ? Either would be a win for Microsoft. Just saying "ya boo sucks ... we're faster than you " isn't.

Would VMware spread disinformation ? Sure they would. These are the people who can title a section "VMware ESXi – The Most Advanced Hypervisor" and in the very next sentence say "VMware ESXi 3.5 is the latest generation of the bare-metal x86 hypervisor that VMware pioneered and introduced over seven years ago.". So a design that's more than 7 years old and wasn't designed to exploit the latest Intel and AMD technology is also the most advanced ? These are the people who can claim "Many VMware ESX customers have achieved uptimes of more than 1,000 days without reboots." which is pretty remarkable when you look at impendent analysis of VMware's patch history. (follow the link and you'll find a quoted interval of every 19 days... 50 sets of missed patches ! Don't tell the boss). 

The Xen and Microsoft architectures rely on routing all virtual machine I/O to generic drivers installed in the Linux or Windows OS in the hypervisor’s management partition. These generic drivers can be overtaxed easily by the activity of multiple virtual machines

When I challenged VMware to find a customer who was in production with the over-commit ratios they claimed, they could only produce one who was thinking about it. So I think I don't think I'm being unfair in calling it BS. Interestingly the post I linked to above repeats that claim.  So I really don't feel bad calling their pronouncement on drivers BS. (I'll wait to see if someone from VMware comes up with a reason why the sum of activity many small VMs is different from one big one. )

Comments
  • PingBack from http://www.ditii.com/2008/07/21/hyper-v-scaling/

  • Microsoft introduced Windows over 23 years ago!  Does this mean that Vista isn't 'advanced'?

  • Nick,. No it doesn't mean that. The things you can identify from Windows 1.0 (Notepad and  Calculator) sure aren't advanced. And the things which make Windows less "advanced" than it might be are all linked to backwards compatibility.

    What is interesting is that newer designs which exploit AMD-V or Intel-VT are said by VMware to be less "advanced". It's a claim that deserves the Tag of BS, wouldn't you say ?

  • It could be said that the virtualisation technology which should be classed as 'more advanced' would be the one with the non-beta management tools!

    The point I was trying to make is that your argument that the date of original release is some how related to how advanced a current product can be classed is totally spurious.  In fact, it could be more generally argued that greater experience and knowledge that is gained from having presence in a market for longer may help deliver a more advanced product and perhaps more crucially, a product that has the feature set that best matches the customer needs.

    VMware runs on processors supporting with AMD-V and Intel-VT and Intel publish figures showing a performance increase on ESX with this technology enabled.  However, they are a hardware requirement for Hyper-V which, although modest, does limit its backward compatibility.

    I know that Microsoft come new to this marketplace and they are trying hard to raise the public awareness but I really think this idea of 'shouting BS' at your competitors is one that neither helps customers nor can be viewed as all constructive.

  • Hi James,

    Excellent article as usual and I fully agree with your summation that some other companies are fine purveyors of BS if nothing else.

    As a matter of interest it would be good to hear a follow on to your article bearing in mind the recent announcement from VMWARE that ESXi will be provided free - http://windowsitpro.com/article/articleid/99830/vmware-announces-q2-08-earnings-makes-esxi-hypervisor-a-free-download.html

    James.

  • Nick, you're saying because the (superior and cheaper) management product for Hyper-V is so new it isn't even released means that the underlying hypervisor technology is less advanced ? that's nonsense.

    You and I seem to be making the same point. The VMware folks post suggests "Most advanced" and "longest established" is somehow linked. That's BS. Generally backwards compatibility holds back development. Hyper-V was designed with those chips in mind, VMware wasn't. Bad choice of words on my part to suggest they don't support AMD-V / Intel-VT.  

    There have been so many instance of folk from VMware going out and telling people things which are simply not true. I think we should call them on that when it happens. People call Microsoft on things like that ... and rightly so.

    James C

    Thanks ... I'm trying to figure out what the VMWare change means in practice. I may just provide a link to what someone from the Hyper-V team ends up posting

  • Nick, you're saying because the (superior and cheaper) management product for Hyper-V is so new it isn't even released means that the underlying hypervisor technology is less advanced ? that's nonsense.

    You and I seem to be making the same point. The VMware folks post suggests "Most advanced" and "longest established" is somehow linked. That's BS. Generally backwards compatibility holds back development. Hyper-V was designed with those chips in mind, VMware wasn't. Bad choice of words on my part to suggest they don't support AMD-V / Intel-VT.  

    There have been so many instance of folk from VMware going out and telling people things which are simply not true. I think we should call them on that when it happens. People call Microsoft on things like that ... and rightly so.

    James C

    Thanks ... I'm trying to figure out what the VMWare change means in practice. I may just provide a link to what someone from the Hyper-V team ends up posting

  • James, I don´t think you´re acting like a IT Pro (much less like an MS IT Pro!!!!), calling things BS and stuff... I´m a MS user proud and through, but VMware is clearly better in TCO and delivery than Hyper-V.

    I hope MS deliver a better virtualization solution in a couple years, but trying to make Hyper-V look better than ESX is like trying to make Vista look as a better choice than XP. Intel dumped Vista from it´s internal usage roadmap in favor of XP, as many other (including me) have done. And we all know this is happening in almost every company.

    I´ll be waiting for Hyper-V2 and Windows 10, ESX and XP are the winners, even if MS doesn´t like it.

  • Juan, that's going to be some wait, as Windows7 isn't going to be around for a bit. Wait for 8 has become the running joke inside MSFT.

    Have you run Hyper-V ? My guess is not, so you're not really in a position to say if it is good, bad or indifferent. Plain and simple fact is that VM won't just go away because Microsoft got into the market which they're used to a share 70% plus. They will charge more than we will, and they will justify it with features we don't have. But if they try to defend their market share by dishonesty then they deserve to get called on it. (We did that once, we got called, and most people would say we deserved it).

    I guess you don't run Vista. Here a simple fact, it does the job better than XP. I held off upgrading the relic that I have at home from XP to Vista and grew so sick of XP (for stuff I couldn't do on work's laptop) that I finally put 2GB of RAM in it and leaves XP in the dust. We know how many companies are buying Vista licences and how many are deploying. The rate for the first year of Vista was the same as the first year of XP - which had much the same complaints, even though it was a small face lift to Win2K. (Hardly anyone shifted to Win2K on the desktop).

    Choosing XP over Vista is choosing the path of lower productivity. You're free to make that choice.

  • James, I'm saying that VMware can currently offer the market a complete solution including hypervisor (now freely available) and management tools.  Microsoft's tools might be absolutely cutting edge but since they aren't out of beta yet, they are currently utterly worthless in a production environment.  Having 'advanced' software is a small achievement and surely having software that is actually 'usable' trumps it?

    Personally, I wouldn't want your job of having to promote the current and relatively green Microsoft virtualisation technologies over your competitors but I agree with Juan that this very negative 'BS' business really does not look at all professional.

  • Nick, to a degree you're right anyone looking to go into full scale production with the Microsoft management tools has a few weeks to wait. Actually we've got real customers who went into production when Hyper-V was still in Beta (it was that good, and that reliable) and who are managing it with the pre-release SCVMM. I would advocate doing that but it does work. Hyper-V is not our first step into virtualization, and we're talking about the second generation of a management tool.

    You seem to have swallowed the VMware book of FUD whole.What we're hearing from customers  is either "We like the price, scalability  performance and Windows integration" or "Come back when you have Live Migration"  

    Various people in VMware seem to be scared by Hyper-V into various kinds of dishonesty. When they lie, we're going to say "that's a lie" End of.

  • James, for once I think I agree with you on one of your points in that I am being pushed far too much VMware FUD.  Just looking at your initial quote about ESXi again:

    “So a design that's more than 7(1) years old and wasn't designed (2) to exploit the latest Intel and AMD technology is also the most advanced(1)(2)?”

    1: You are suggesting that the date of original release of a product is related to how advanced a current product can be classed.  You subsequently and very eloquently called the relationship ‘BS’ after I tried turning the relationship between age/how advanced to the 23 year old Microsoft ‘Windows’ family of products.

    2: You are suggesting that ESXi doesn’t support AMD-V / Intel-VT.  It does and in a subsequent post you owned up to this phrase being a ‘bad choice of words’.

    So, by your own admissions you’ve not really left a whole lot of credibility or substance in your original statement.  I’m a Microsoft customer and I’ve had a go at engaging in some very reasoned discussion with you but you’ve just rudely dismissed me as having ‘swallowed the VMware book of FUD whole’. Well, I for one can think I can see where the book of VMware FUD is being authored.

  • If Hyper-V is as good as ESX, why did Microsoft explicitly change licensing rules to prevent users of ESX from legally taking advantage of the vmotion capabilities?

    I'm referring to the license rewrite telling me that I can only use vmotion once every 6 months and can never move the virtual machine back to the original hardware. If I bought a Windows Server license for each VM what does MS benefit other than to force a less realistic product feature comparison via arbitrary restrictions?

    And while I'm at it, licensing multiple desktop VMs ONLY with SA agreements (because otherwise SA agreements are valueless?) Come on, this is the act of a company running scared, not a company that is keeping ahead of the technology curve.

  • Nick. I can only guess why you want to twist what say, quite so far.  I made a minor point that VMware claim to be the oldest and the most advanced, and those too tend to be exclusive.

    VMware was designed before processors supported virtualization, and support for latest CPU tricks gets fudged into each new release. Xen and Hyper-V were designed to exploit those. There is a difference.

    It's no accident that the reference design on which both XEN and Hyper-V came out of the academic world who rejected the VM approach.

    Sean. I **think** you're misreading the licence. As I understand it [and the following is not official licensing advice and conveys no rights ...] Each *box* CAN be licensed for number of instances of Windows it runs. Enterprise gives you 4 working instances. Those licences were never free floating, and never divisible. So if you had 8 VMs on one box you could give it 2 Enterprise licences. If you move a VM to another box it needs a licence on that box. If you move 4 VMs back and forth between boxes you can't keep flipping the licence back and forth. However if you buy single licenses for standard server and assign them to a VM you can move the VM round just like you'd move a server box.

    We gave the 4 working instances away as part of server 2003 r2 back in 2005 and we had to keep working against VM FUD which said we didn't let people use it on non MS platforms. I may be wrong but I don't think we have changed anything in the License agreement since then, just clarified what it means.

    Alternatively you assign a licence to a VM

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment