MMS 2010 Labs: Powered by Hyper-V, System Center & HP...

re: MMS 2010 Labs: Powered by Hyper-V, System Center & HP...

Patrick — Fri, 28 May 2010 23:58:52 GMT

@Frank Pinto: there’s no public URL with a detailed description. Let’s take this offline and we can help you. Please click “email blog author” [find it under This Blog, left navigation bar], and share your email address and cut/paste the question. Either Jeff or Ronald will email you back.

re: MMS 2010 Labs: Powered by Hyper-V, System Center & HP...

frankpintosr — Thu, 27 May 2010 16:10:19 GMT

I was at MMS 2009 and 2010 an was very impressed with the lab setup and performance this year. And to add to previous comments, the dashboard was fantastic.

So, here is my question... how/where/when can I get a detailed descritpion of the setup for the lab? I think this is something I could use in a much smaller scale. I am a Senior Consult and part of what I do is training IT staff on new technology we are implementing. It would be amazing to add something like this in a very small and portable setup. And instead of using thin clients, utilize VPCs on their own hardware pushed with MDT2010 or something. Anyway, you have really got my gears turning here, lol.

Great Job, I am looking forward to hearing back from you.

Regards,

Frank Pinto

re: MMS 2010 Labs: Powered by Hyper-V, System Center & HP...

Ronald Beekelaar — Thu, 13 May 2010 22:22:05 GMT

Hi Weston. Yes, of course we can let you know what happened on Monday afternoon.

On Monday morning, everything was fine, but then during the day several labs became unresponsive, and some even failed to start. So if you were there on Monday afternoon, you experienced that.

This was caused by a bug in our Web code. After we fixed that on Monday evening, all was working well until the end of the show.

The technical explanation is as follows:

The bug was a "logical mistake" in the way our Web site handled the load-balancing between multiple Hyper-V servers. When all 100 clients from an instructor-led lab session started at the exact same time, too many clients were assigned to the same Hyper-V server.

This caused some labs to fail (out-of-memory), and not close down correctly. So during the day some running lab VMs where left behind, and that made the problem more noticeable in the afternoon.

When we changed the load-balancing logic in the Web site to be a little bit smarter - all was fine the rest of the time. So this was NOT related to disk IO, or scaling, merely a logical mistake when handling 100 concurrent lab-start requests at the exact same time.

> Looking forwared to MMS 2011!

So are we! Hope to see you there!

Cheers,

Ronald Beekelaar

XB Velocity

re: MMS 2010 Labs: Powered by Hyper-V, System Center & HP...

Weston — Thu, 13 May 2010 20:21:51 GMT

Just curious what the issue was with the Labs on Monday? I was there as well and the labs on Monday afternoon were unusable. Seemed like the issue got ironed out but wondering if you could elaborate on what happened? Was that the disk IO you mentioned in the previous comments? If a high powered san was used would that have alleviated the issue?

Looking forwared to MMS 2011!

re: MMS 2010 Labs: Powered by Hyper-V, System Center & HP...

Mr EBE — Tue, 11 May 2010 07:09:40 GMT

Impressive stuff, indeed. Now that the show is finished, can you send me one of your rack (with the built-in cheap servers of course, don't bother removing them) ?

With all my thankx (shipping cost should stay low as I live in switzerland)

Best greetings

re: MMS 2010 Labs: Powered by Hyper-V, System Center & HP...

Ronald Beekelaar — Thu, 06 May 2010 17:59:31 GMT

Hi aknab. Thanks for bringing this up. You are absolutely right!

My team (XB Velocity) set up the lab system at MMS 2010. One of the things that we noticed is that for during some instructor-led labs, high-disk intensity activities (running an OS deployment as 10 concurrent labs, etc) may occur at exactly the same time, on the same server. That may impact other labs on the same server.

So one of the improvements we are making at next events, is to spread out multiple copies of the same lab over more servers. This will spread out the disk intensive task more, rather than occurring 10 times in-sync on the same Hyper-V server.

BTW - Glad that you liked the dashboard display. That got some good comments <g>.

Cheers,

Ronald Beekelaar

re: MMS 2010 Labs: Powered by Hyper-V, System Center & HP...

aknab — Thu, 06 May 2010 17:11:30 GMT

I was at MMS and was very impressed with the Lab setup, however the labs were very unresponsive. That did not make a whole lot of sense until I ran a perfmon and it indicated a disk bottleneck(1-3 Mbytes/sec max). Since you are not using a san I have no idea why this occurred. Just thought you should check it out but I got a kick out of the dashboards you had running!