Virtualization Nation,In my last blog, we announced the RTM (Release to Manufacturing) of Service Pack 1 for Windows 7 and Windows Server 2008 R2 SP1. The bits will be available for download on Feb. 22, so mark your calendars.
A frequent follow-up question to hit my inbox was from folks interested in a list of documented changes included in Windows 7 and Windows Server 2008 R2 SP1 in addition to Dynamic Memory and RemoteFX.
No problem.
Here’s the link to the documentation for Windows 7 and Windows Server 2008 R2 SP1 (KB976932). This KB includes:
While the current version posted is for the Service Pack 1 Release Candidate, the final version will be available shortly for the RTM version.VMware and ASLR Follow-Up
In my last blog, I discussed the importance of Address Space Layout Randomization (ASLR) as an effective, transparent security mitigation built-into Windows 7. I noted that independent security analysts wholeheartedly agree on the importance of ASLR. I also stated we have serious concerns that VMware was recommending customers disable ASLR to achieve better density.Following that blog post, we were contacted by Jeff Buell from VMware.
From Jeff Buell, Perf Engineering at VMware
I'm from the performance engineering team at VMware. We take both performance recommendations and security very seriously. As you state, ASLR is a good security feature. VMware has never recommended disabling it. If you have a reference saying otherwise, I'd love to see it.
First, let me say thank you to Jeff Buell for his swift response. I’m glad to see that Microsoft and VMware Engineering agree that ASLR is a good security feature and that disabling ASLR is a terrible suggestion. Jeff appears to be concerned and willing to rectify this situation. Again, thank you Jeff. Here are the specifics.
Looks Like It Started Here…
It appears that the suggestion to disable ASLR began right here on VMware’s public blog page.
http://blogs.vmware.com/view/2009/04/vista-and-vmware-view.html
The post casually mentions that disabling ASLR will “lower overall security,” and then continues to make things worse by telling people to disable NX and DEP, two additional security mitigations. Because of this post, others picked up on this recommendation (such as in VMware’s community forums) and promoted this idea without anyone from VMware disputing this unfortunate suggestion:
http://communities.vmware.com/message/1294525#1294525
At first, I thought these were isolated incidents, but then I started receiving regular inquiries from customers who said they were considering a VDI deployment and specifically asking if Microsoft had a recommendation or support stance regarding ASLR. Considering the fact that ASLR is transparent and you have to go out of your way to disable it (you have to be admin and then go to the Registry), I knew this wasn’t isolated anymore.
Finally, at VMworld 2010 in Europe, VMware Director of Product Marketing, Eric Horschman, delivered session TA8270 titled, Get the Best VM Density From Your Virtualization Platform.
In this session, a slide was presented with the following:
Best practices> Blame storage first - avoid bottlenecks> Upgrade to vSphere 4.1 for memory compression> Install VMware tools in guest OSes to enable ballooning> Protect your critical VMs> Add VMs until “active” memory overcommit is reached> Allow DRS to balance VMs across your cluster
Advanced techniques
> Use flash solid state disks for ESXi swapfile datastore (for overcommitted hosts)> Adjust HaltingIdleMsecPenalty (KB article 1020233)> Consolidate similar guest OSes and applications to assist Transparent Page Sharing> Disable ASLR in windows 2008/Windows 7 guests for VDI workloads
When a VMware Director is promoting such poor advice, we were concerned our customers were putting themselves in undue risk and wanted to clearly articulate the Microsoft position. There is an apparent disconnect between VMware engineering and marketing on this topic, and I’m glad to see the engineering team speak out.
Again, my thanks to Jeff Buell from VMware Engineering for his quick response to this matter. I’m going to assume that VMware will clarify their position internally and appropriately message their position externally by fixing these external links. I’d be relieved to see VMware no longer recommend users disable fundamental security mitigations, such as ASLR, any further.
In my next blog, I’ll discuss some points you should consider make when determining what guest OS to deploy for VDI.
Cheers,
Jeff WoolseyGroup Program Manager, Hyper-VWindows Server & Cloud
So you heard about the Hyper-V Cloud Fast Track program and wonder … what exactly is it? Is it marketecture or is there some meat to it? What do these “pre-architected, pre-validated” solutions consist of, and how were those decisions arrived at? What was the architecture and validation methodology? Well my friend, this post is for you.
Myself and David Ziembicki authored the Hyper-V Cloud Fast Track Reference Architecture and Validation Guide used to align Microsoft and OEM partners on a common architecture and for OEMs to re-use and expand upon for their own Reference Architectures. Dave and I are Solution Architects within Microsoft Services in the US Public Sector organization.
A few details to get out of the way: First, each OEM (HP, Dell, IBM, Hitachi, Fujitsu, NEC) brings something unique to the table. Each OEM partner will be jointly publishing with Microsoft their Hyper-V Cloud Fast Track Reference Architecture, which will detail the hardware specifications, configurations, detailed design elements, and management additions. Available to you right now are some great resources such as solution briefs, a new white paper, and Fast Track partner web sites. In this post I will share with you the common architecture elements that apply to all program partners and how those decisions were made.
Next, I’d like to direct you to the Private Cloud TechNet Blog where I have detailed the Principles and Concepts which underlay the architecture of this program. Now, those principles are actually pretty lofty goals and the program will address more and more of them over time. A brief preview of the concepts is listed below. I feel it’s important to provide a glimpse of them now because they are what Hyper-V Cloud Fast Track aims to achieve. Please reference the post for deeper insight on these.
Private Cloud Concepts
Resiliency over Redundancy Mindset – This concept moves the high-availability responsibility up the stack from hardware to software. This allows costly physical redundancy within the facilities and hardware to be removed and increases availability by reducing the impact of component and system failures.
Homogenization and Standardization – by homogenizing and standardizing wherever possible within the environment, greater economies of scale can be achieved. This approach also enables the “drive predictability” principle and reduces cost and complexity across the board.
Resource Pooling – the pooling of compute, network, and storage that creates the fabric that hosts virtualized workloads.
Virtualization – the abstraction of hardware components into logical entities. I know readers are of course familiar with server virtualization, but this concept speaks more broadly to benefits of virtualization across the entire resource pool. This may occur differently with each hardware component (server, network, storage) but the benefits are generally the same, including lesser or no downtime during resource management tasks, enhanced portability, simplified management of resources, and the ability to share resources.
Fabric Management – a level of abstraction above virtualization that provides orchestrated and intelligent management of the fabric (i.e., datacenters and resource pools). Fabric Management differs from traditional management in that it understands the relationships and interdependencies between the resources.
Elasticity – enables the perception of infinite capacity by allowing IT services to rapidly scale up and back down based on utilization and consume demand
Partitioning of Shared Resources – While a fully shared infrastructure may provide the greatest optimization of cost and agility, there may be regulatory requirements, business drivers, or issues of multi-tenancy that require various levels of resource partitioning
Cost Transparency – provides insight into the real costs of IT services enabling the business to make informed and fair decisions when investing in new IT applications or driving cost-reduction efforts.
Hyper-V Cloud Fast Track Architecture Overview
With the principles and concepts defined we took a holistic approach to the program thinking first about everything that would be ideal to achieve an integrated private cloud and pairing down from there to what now forms the first iteration of the offering. As stated, future versions will address more and more of the desired end-state.
Scale Unit
Scale Units represents a standardized unit of capacity that is added to a Resource Pool. There are two types of Scale Unit; a Compute Scale Unit which includes servers and network, and a Storage Scale Unit which includes storage components. Scale Units increase capacity in a predictable, consistent way, allow standardized designs, and enable capacity modeling.
Server Hardware
The server hardware itself is more complex that it might seem. First, what’s the ideal form-factor? Rack-mount or Blade? While we certainly have data that shows blades have many advantages for virtualized environments, they also can add cost and complexity for smaller deployments (4-12 servers). This is one decision where we provided guidance and experience on, but ultimately left the decision to the OEM as to when blades made sense for their markets. Most OEMs who have both blade and rack-mount options and will be offering both through this program.
For CPU, all servers will have a minimum of 2-socket, quad-core processors yielding 8 logical processors. Of course, many of the servers in the program will have far more than 8 LPs, likely 12-24 will be most common as that’s the current price/performance sweet-spot. BTW - Hyper-V supports up to 64 LPs. The reason for this is that although the supported ratio of Virtual Processors to Logical Processors is 8:1, real-world experiences with production server workloads have shown more conservative average ratios. Based on that we concluded 8 LPs should be the minimum capacity starting point.
Storage
Storage is where, for me anyway, things begin to get really interesting. There are just so many exciting storage options for virtualized environments these days. Of course, it’s also a design challenge: which features are the highest priority and worth the investment? We again took a holistic approach and then allowed the partner to inject their special sauce and deep domain-expertise. Here’s the list of SAN storage features we targeted for common architecture criteria:
o High Availability
o Performance Predictability
o Storage Networking
o Storage Protocols
o Data De-duplication
o Thin Provisioning
o Volume Cloning
o Volume Snapshots
o Storage Tiering
o Automation
One of the really cool advantages of this program is that it allows for multiple best-of-breed private cloud solutions to emerge taking advantage of each vendor’s strength. You can only find this in a multi-vendor, multi-participant program.
On the Hyper-V side we provided common best-practices for Cluster Shared Volume configuration, sizing, and management as well as considered such things as MPIO, Security, I/O segregation, and more.
Network
Networking presents several challenges for Private Cloud architectures. Again here we find a myriad of choices from the OEMs and are able to leverage the best qualities of each where it makes sense. However, this is an area where we sometimes find IT happening for IT’s sake (i.e. complex, advanced networking implementations because they are possible and not necessarily because they are necessary to support the architecture). We need to look at the available products and features and only introduce complexity when it’s justified as we all know increased complexity often brings with it increased risk. Some of those items include:
o Networking Infrastructure (Core, Distribution, and Access Switching)
o Performance Predictability and Hyper-V R2 Enhancements (VMQ, TCP Checksum Offload, etc.)
o Hyper-V Host Network Configuration
o 802.1q VLAN Trunks
o NIC Teaming
NIC Teaming in particular is one of those items that can be tricky to get right being there are different vendor solutions each with potentially different features and configuration options. Therefore it’s an example of a design element that benefits greatly from the Hyper-V Cloud Fast Track program taking all the guesswork out of NIC Teaming providing the best-practice configuration tested and validated by both Microsoft and the OEM.
Private Cloud Management
Let’s face it, cloud computing places a huge dependency on management and operations. Even the most well designed infrastructure will not achieve the benefits promised by cloud computing without some radical systems management evolution.
Again leveraging the best-of-breed advantage, a key element of this architecture lies in that the management solution may be a mix of vendor software. Notice I said may. That’s because a vendor who is a big player in the systems management market may have chosen to use their software for some layers of the management stack while others may have chosen to use an exclusively Microsoft solution consisting of System Center, Forefront, Data Protection Manager, etc. I will not attempt to cover each possible OEM-specific solution. Rather, I just want to point out that we recognize the need and benefit of OEMs being able to provide their own elements of the management stack, such as Backup and Self-Service Portal. Some are, of course, essential to the Microsoft virtualization layer itself and are non-replaceable such as System Center Virtual Machine Manager and Operations Manager. Here is a summary of the management stack included:
o Microsoft SQL Server
o Microsoft System Center Virtual Machine Manager and Operations Manager
o Maintenance and Patch Management
o Backup and Disaster Recovery
o Tenant / User Self-Service Portal
o Storage, Network and Server Management
o Server Out of Band Management Configuration
The Management layer is so critical and really is what transforms the datacenter into a dynamic, scalable, and agile resource enabling massive capex and opex cost reduction, improved operational efficiencies, and increased business agility. Any one of these components by themselves is great, but it’s the combination of them all that qualify it as a private cloud solution.
Summary
There are several other elements I would love to delve into such as Security and Service Management, but this post could go for quite a while. I’ll leave the remainder for the Reference Architecture Whitepaper which we just published, as well as the OEM-specific Reference Architectures published by them.
I hope you found this article useful and that it sheds some light on the deep and broad collaborative effort we have embarked upon with our partners. Personally, I am very happy that this program was created and am confident it will fill a great need emerging in datacenters everywhere.
Adam Fazio, Solution Architect, Microsoft
Virtualization Nation,
On behalf of the Windows Server and Cloud teams at Microsoft, I’m pleased to announce that today we released Service Pack 1 for Windows Server 2008 R2 and Windows 7 – adding two new virtualization capabilities: RemoteFX and Dynamic Memory. SP1 will be made generally available for download on February 22. To learn more about RemoteFX, take a look at Michael’s Kleef’s blog. I’ll cover Dynamic Memory and a few other updates you’ll want to understand.
Let’s start with Dynamic Memory. An enhancement to Hyper-V R2, Dynamic Memory pools all the memory available on a physical host. Dynamic Memory then dynamically distributes available memory, as it is needed, to virtual machines running on that host. Then with Dynamic Memory Balancing, virtual machines will be able to receive new memory allocations, based on changes in workload, without a service interruption. In short, Dynamic Memory is exactly what it’s named (I wrote a six part blog series on Dynamic Memory here: Part 1, 2, 3, 4, 5, and 6).
Why is Dynamic Memory so important?
High praise from the folks over at brianmadden.com:
I do think that, looking at memory management from a VDI perspective, Hyper-V fits the bill just as well as ESX does, if not better.
Is Hyper-V Dynamic Memory any good for VDI? Definitely! I love it.
Making the most of Dynamic Memory can really be worth your while. In fact Microsoft has seen improvements of up to 40% (!) in density for VDI workloads.
With VMware it's also easier to oversubscribe the physical memory of the host (note how I didn't use the word overcommit!) and I think that's a risk in most current VDI deployments. No matter how you slice it or dice it, when RAM is oversubscribed it introduces a higher probability of paging. This in return means a huge increase in IOPS. I guess it should go without saying that this is something you should avoid at all costs in VDI environments.
Dynamic Memory takes Hyper-V to a whole new level. Dynamic Memory lets you increase virtual machine density with the resources you already have—without sacrificing performance or scalability. Ultimately it helps customers get the most bang for their technology bucks, which is a critical part of Microsoft’s virtualization and infrastructure strategy. Without that, you’ll keep pouring money into complex solutions you might not need.
Dynamic Memory and Virtual Desktop InfrastructureAlong the lines of determining what’s critical, in our lab testing, with Windows 7 SP1 as the guest operating system in a Virtual Desktop Infrastructure (VDI) scenario, we saw a 40% increase in density from Windows Server 2008 R2 RTM to SP1. We achieved this increase simply by enabling Dynamic Memory. More importantly, this increase in density didn’t require the user to make changes to the guest operating system at the expense of security, as is the case with competitive offerings.
Full stop. I want to reemphasize that last sentence.
Let me explain. In our testing of Dynamic Memory, we’ve also been reviewing VDI deployments and best practice guidance offered by VMware and others. We’ve seen some interesting ideas, but unfortunately we’ve also seen some questionable (if not terrible) suggestions such as this one that we’ve heard from a number of VMware folks: Disable Address Space Layout Randomization (ASLR).
The Importance of ASLRASLR is a feature that makes it more difficult for malware to load system DLLs and executables at a different location every time the system boots, as a way to find out where APIs are located. Early in the boot process, the Memory Manager picks a random DLL image-load bias from one of 256 64KB-aligned addresses in the 16MB region at the top of the user-mode address space. As DLLs that have the new dynamic-relocation flag in their image header load into a process, the Memory Manager packs them into memory starting at the image-load bias address and working its way down.
ASLR is an important security protection mechanism introduced in Windows Server 2008 and Windows Vista. ASLR has helped protect customers from malware and has been further improved in Windows Server 2008 R2 and Windows 7. Best of all, you don’t need to do anything to take advantage of ASLR: It’s enabled by default, it’s transparent to the end user and it just works. In fact, third parties agree that Windows 7 has taken another massive leap forward:
Sophos Senior Security Advisor Chet Wisniewski says "ASLR was massively improved in Windows 7. This means that libraries (DLL’s) are loaded into random memory addresses each time you boot. Malware often depends on specific files being in certain memory locations and this technology helps stop buffer overflows from working properly."
For the record, Microsoft does not recommend disabling ASLR. So, why would anyone recommend disabling ASLR? Read on.
Project VRCLet’s take a look at a report performed by an independent third party, Project Virtual Reality Check (VRC).
The folks at Project VRC have developed their own test methodology and have been working in the industry to better understand the complexities of virtual desktop and remote desktop session capacity planning and deployment. In their latest tests, “Project VRC Phase III (here),” the Project VRC team specifically tested enabling and disabling ASLR to see how it impacted VMware’s density. So what did they find?
Project VRC Phase III, Page 35It must be noted that Project VRC does not blindly recommend disabling ASLR. This is an important security feature, and it is enabled by default since Windows Vista and Windows [Server] 2008 (Windows XP and Windows [S]erver 2003 do not support ASLR). However, with VDI workloads, the impact could be potentially larger. Every desktop session is running an individual desktop OS instance. In comparison to Terminal Services, a VDI workload runs a magnitude of OS’s more to serve desktops to end-users. Potentially the performance impact of ASLR could be larger.
Project VRC evaluated the impact of ASLR on a Windows 7 desktop workload (120 VM’s pre-booted, 1GB memory, 1vCPU per VM, 2GB Page file fixed, VRC optimizations, ESX 4.0 Update 2, HIMP=100):
Figure 1: VMware overcommit doesn’t work well with ASLR
By disabling ASLR, the VSImax score was 16% higher. In comparison to the 4% increase witnessed on Terminal Services, the increase in capacity with Windows 7 VDI workloads is significantly higher. This does not come as a total surprise: the amount of VM’s running is also significantly higher. Although it is difficult to generally recommend disabling ASLR, the impact on Windows 7 is considerable.
In short, VMware recommends disabling a fundamental security feature in Windows because their Memory Overcommit doesn’t work well with ASLR. Not a good idea. Let’s see how Hyper-V R2 SP1 Dynamic Memory fares.
Hyper-V R2 SP1 Dynamic Memory & ASLRWe decided to perform similar tests (not identical so please don’t make a direct comparison with the VMware data; the hardware was different) using the same Project VRC Phase III test methodology. The point of this test was to compare running Windows 7 as a Hyper-V guest with and without ASLR enabled in the guest OS and to compare the delta of running with ASLR enabled. With VMware there was a considerable delta. What about with Hyper-V?
Here are the results:
Figure 2: Hyper-V works great with ASLR
You can see that with Hyper-V and ASLR, the results are virtually identical whether ASLR is on or off. That’s because Dynamic Memory was designed from the ground up to work with ASLR and other advanced memory technologies. You won’t hear anyone from Microsoft suggest you turn off ASLR.
Personally, I am convinced Dynamic Memory is a big step forward. I say this because it literally changes the way I create and deploy virtual machines (VMs). I assign the VM its startup value and then I simply don’t worry any more. Dynamic Memory effectively solves the problem of “how much memory do I assign to my server?” as discussed here. The approach is both efficient and elegant.
I should also point out that Hyper-V Dynamic Memory will be available in Microsoft Hyper-V Server 2008 R2 SP1, the free download of the stand-alone hypervisor-based virtualization product.
In addition to SP1, we’ve been very busy with our virtualization technology updates and want to be sure you’re aware of the latest:
Higher Virtual Processor to Logical Processor Ratios: If you’re running Windows Server 2008 R2 SP1 and running Windows 7 as the guest, we’ve upped the ratio of virtual processors to logical processor from 8:1 to 12:1. This is simply more goodness for VDI deployments. This change is documented here.
Higher Cluster Density and Limits: Back in June 2010, the Microsoft Failover Cluster team upped the support limit to 384 virtual machines per node to match the Hyper-V maximum of up to 384 virtual machines per server. In addition, the overall number of running VMs per cluster has been bumped to 1000 VMs in a cluster. Read more here.
New Linux Integration Services: Back in July 2010, we released new Linux Integration Services, which added support for more Linux distributions and new capabilities, including:
And while this was happening, we’ve been powering our own tradeshows (examples: MMS 2010, TechEd 2010) with Hyper-V and System Center—with tremendous benefits.
=====================================================================
P.S. Here are the links with descriptions to the six part series titled Dynamic Memory Coming to Hyper-V, and an article detailing 40% greater virtual machine density with DM.
Part 1: Dynamic Memory announcement. This blog announces the new Hyper-V Dynamic Memory in Hyper-V R2 SP1. It also discusses the explicit requirements that we received from our customers. http://blogs.technet.com/virtualization/archive/2010/03/18/dynamic-memory-coming-to-hyper-v.aspx
Part 2: Capacity Planning from a Memory Standpoint. This blog discusses the difficulties behind the deceptively simple question, “how much memory does this workload require?” Examines what issues our customers face with regard to memory capacity planning and why. http://blogs.technet.com/virtualization/archive/2010/03/25/dynamic-memory-coming-to-hyper-v-part-2.aspx
Part 3: Page Sharing. A deep dive into the importance of the TLB, large memory pages, how page sharing works, SuperFetch and more. If you’re looking for the reasons why we haven’t invested in Page Sharing, this is the blog. http://blogs.technet.com/virtualization/archive/2010/04/07/dynamic-memory-coming-to-hyper-v-part-3.aspx
Part 4: Page Sharing Follow-Up. Questions answered about Page Sharing and ASLR and other factors to its efficacy. http://blogs.technet.com/b/virtualization/archive/2010/04/21/dynamic-memory-coming-to-hyper-v-part-4.aspx
Part 5: Second Level Paging. What it is, why you really want to avoid this in a virtualized environment and the performance impact it can have. http://blogs.technet.com/b/virtualization/archive/2010/05/20/dynamic-memory-coming-to-hyper-v-part-5.aspx
Part 6: Hyper-V Dynamic Memory. What it is, what each of the per virtual machine settings do in depth and how this all ties together with our customer requirements. http://blogs.technet.com/b/virtualization/archive/2010/07/12/dynamic-memory-coming-to-hyper-v-part-6.aspx
Hyper-V Dynamic Memory Density. An in depth test of Hyper-V Dynamic Memory easily achieving 40% greater density. http://blogs.technet.com/b/virtualization/archive/2010/11/08/hyper-v-dynamic-memory-test-for-vdi-density.aspx