Information and announcements from Program Managers, Product Managers, Developers and Testers in the Microsoft Virtualization team.
Preamble: The point of this series, and the spirit in which it is written, is to take a holistic approach at the issues facing our customers, discuss the complexities with regard to memory management and explain why we’re taking the approach we are with Hyper-V Dynamic Memory. This isn’t meant to criticize anyone or technology, rather to have an open and transparent discussion about the problem space.
In the past few blogs we’ve covered Page Sharing and Second Level Paging. Today, let’s dig into what we’re delivering with Hyper-V Dynamic Memory in Windows Server 2008 R2 SP1 as well as our free hypervisor Microsoft Hyper-V Server 2008 R2 SP1. So what is Dynamic Memory?
Dynamic memory is an enhancement to Hyper-V R2 which pools all the memory available on a physical host and dynamically distributes it to virtual machines running on that host as necessary. That means based on changes in workload, virtual machines will be able to receive new memory allocations without a service interruption through Dynamic Memory Balancing. In short, Dynamic Memory is exactly what it’s named.
Let’s dive in an explain how all this works starting with the new Dynamic Memory settings. Here are the new settings available on a per virtual machine basis. Here’s a screenshot:
Dynamic Memory In Depth
With Hyper-V (V1 & R2), memory is statically assigned to a virtual machine. Meaning you assign memory to a virtual machine and when that virtual machine is turned on, Hyper-V allocates and provides that memory to the virtual machine. That memory is held while the virtual machine is running or paused. When the virtual machine is saved or shut down, that memory is released. Below is a screenshot for assigning memory to a virtual machine in Hyper-V V1/R2:
With Hyper-V Dynamic Memory there are two values: Startup RAM and Maximum RAM and it looks like this:
Startup RAM is the initial/startup amount of memory assigned to a virtual machine. When a virtual machine is started this is the amount of memory the virtual machine will be allocated. In this example, the virtual machine will start with 1 GB.
The Maximum RAM setting is the maximum amount of memory that the guest operating system can grow to, up to 64 GB of memory (provided the guest OS supports that much memory). Based on the settings above, here’s an example of what the memory allocation could look like over a workday...
As you can see, the workload is dynamically allocated memory based on demand.
Next, let’s look at the Memory Buffer.
Memory Buffer: In one of the earlier blogs posts in this series, we discussed the complexity of capacity planning in terms of memory. To summarize, there is no “one size fits all” answer for every workload as deployments can vary based on scale and performance requirements. However, one consistent bit of feedback was that customers always felt more comfortable by providing additional memory headroom ‘just in case.’
We completely agree.
The point being you want to avoid a situation where a workload needs memory and Hyper-V has to start looking for it. You want some set aside memory as buffer for these situations, especially for bursty workloads.
The Dynamic Memory buffer property specifies the amount of memory available in a virtual machine for file cache purposes (e.g. SuperFetch) or as free memory. The range of values are from 5 to 95. A target memory buffer is specified in percentages of free memory and is based on current runtime memory usage. A target memory buffer percentage of 20% means that in a VM where 1 GB is used, 250 MB will be ‘free’ (or available) ideally for a total amount of 1.25 GB in the virtual machine. By default, Hyper-V Dynamic Memory uses a default buffer allocation of 20%. If you find this percentage is too conservative or not conservative enough, you can adjust this setting on the fly while the virtual machine is running without downtime.
This takes us to the last Dynamic Memory setting, Memory Priority.
Memory Priority: By default, all virtual machines are created equal in terms of memory prioritization. However, it’s very likely you’ll want to prioritize memory allocation based on workload. For example, I can see a scenario where one would give domain controllers greater memory priority than a departmental print server. Memory Priority is a per virtual machine setting which indicates the relative priority of the virtual machine's memory needs measured against the needs of other virtual machines. The default is set to ‘medium’. If you find that you need to adjust this setting, you can adjust this setting on the fly while the virtual machine is running without downtime.
Dynamic Memory Works Over Time With A Few VMs…
I’ve explained the per VM settings and shown how this would work with a single virtual machine, but how does Dynamic Memory work with multiple virtual machines? Below is an example to show just how Dynamic Memory works. I’ve kept this example simple on purpose to avoid confusion. Let’s assume I have a small server with 8 GB of memory. I’m going to run three virtual machines, one from Finance, Sales and Engineering. Each virtual machine is given the same settings: Startup RAM = 1 GB and Maximum RAM = 4 GB. With these settings, each virtual machine will start 1 GB and can grow up to 4 GB as needed.
Virtual Machine Start. On the left graphic below, you can see three virtual machines starting. Each virtual machine is consuming 1 GB of memory for Startup RAM. On the right graphic below, you can see the total amount of memory being used in the entire system ~3 GB.
15 minutes later. The Finance VM is running reports while the Engineering VM starts an analysis job. With Dynamic Memory, the Finance VM is allocated 3 GB of memory, the Engineering VM is allocated 2 GB of memory while the Sales VM remains at 1 GB. System wide, the server is now using 6 GB of its 8 GB or 75% of the total physical memory.
30 minutes later. The Finance VM is running reports while the Engineering VM starts an analysis job. With Dynamic Memory, the Finance VM is allocated 2 GB of memory, the Engineering VM is allocated 3.5 GB of memory while the Sales VM remains at 1 GB and a fourth VM, Service VM is started using 1 GB of memory. System wide, the server is now using 7.5 GB of its 8 GB of memory for VMs. At this point the server is fully allocated in terms of memory and is using its memory most efficiently.
At this point, the question I’m always asked is, “What now? What if a virtual machine still needs more memory? Does the parent start paging?”
At this point, Dynamic Memory will attempt to reclaim pages from other virtual machines. However, in the absolute worst case where no free pages are available, the guest operating system will page as needed, not the parent. This is important because the guest operating system knows best what memory should and shouldn’t be paged. (I covered this back in Part 5...) Finally, when free memory does become available from other virtual machines, Dynamic Memory will move memory as needed.
Over-Subscription & the CPU Analogy
One argument we routinely hear is that there’s nothing wrong with over-subscription. Customers tell us that they take a bunch of physical servers, virtualize them and run the server with over-subscribed CPUs without issue, so why is this an issue with memory?
Great analogy, wrong conclusion.
Example 1: Suppose you are running 8 physical servers at 10% utilization, virtualize them and run those 8 virtual machines on a single server for a total of ~85% utilization. In this example, you’re not over-subscribing the CPU and the server still has 15% CPU headroom.
Over-subscription is this…
Example 2: Suppose you are running 8 physical servers at 50% utilization, virtualize them and run those 8 virtual machines on a single server. The single server would max out at 100% utilization, but because the workloads require ~400% utilization, performance would be terrible. What would you do? Move virtual machines to other servers of course to avoid over-subscription. In short, what you really want to do is maximize resource utilization to get the best balance of resources and performance.
That’s exactly what we’re doing with Hyper-V Dynamic Memory.
Customer Requirements & Dynamic memory
When it comes to virtualization and memory, virtualization users have repeatedly provided the following requirements:
You got it. Here’s why we’ve chosen the path we have with Dynamic Memory.
Principal Group Program Manager
Windows Server & Cloud, Virtualization
P.S. Here are the links to all of the posts in this blog series:
Nice article. May i know how to get 7.5GB in 30 minute. The scenario same as 15 minute. Thanks
I'm scripting virtual machine creation and would like to use dynamic memory feature. Are WMI specifications of DM available ?
Does Hyper-V use large pages by default, or is additional configuration required?
While we're at it, it appears that the Dynamic Memory configuration is in the Hyper-V Manager console, but not in the SCVMM console. Is there some update for SCVMM to allow access to this information, or am I going to have to connect to each Hyper-V host one at a time?
So how is it implemented? Via hot-add of memory, or a balloon driver within the guest?
Can we get some clarification on the question above posted by OtherKevin? We use SCVMM for all of our Hyper-V management tasks. How will this new feature show up in the SCVMM Console?
Yaniv, Ben Armstrong gave a presentation at TechEd about Dynamic Memory that was far more illuminating that anything I've seen before or since. You can catch it here:
@OtherKevin / The realist:
The question is when...
I'm sure it is a matter of "when", but that's not much help is it? If you have a large Hyper-V deployment you will be using SCVMM to manage it. If you have a large Hyper-V deployment the odds are also good that you were waiting with baited breath for Dynamic Memory so that you can get higher VM density and more efficient use of resources. It's unbelievable that Microsoft would roll out new features to the virtualization product that are not supported in their virtualization management product AND not say a word about when it would be. Can you imagine vmWare rolling out new features in vSphere that were not supported in vCenter? Microsoft (and the rest of the world) would be rolling on the floor laughing at them.
This is an issue that I have brought up every time that I've met with anyone involved in the virtualization program at Microsoft (guys like Bryon Surace) and I never get a response other than "yeah". The Hyper-V and SCVMM teams don't march in lockstep, and what updates they do release are too far apart. Microsoft's biggest competitor in the virtualization space has been doing a major and a minor release every year Microsoft manages to eke out one new feature once a year. Heck, Dynamic memory was originally announced and slated to be a feature in the R2 release but got pushed back a year. Should we believe that nobody on the SCVMM team knew that it was eventually coming?
OtherKevin, Microsoft has already announced update for SCVMM 2008 R2 to support Dynamic Memory and RemoteFX in MMS 2010 and Tech ED 2010.
You can review my teched session and move to 64:68 Mins
Now the question is when its going to be released?, I guess we have to wait for a word from Product Group.
There is no date from MS as to when SP1 will be supported - it's just a beta - so it's unrealistic to expect detailed dates about the minor update needed to SCVMM to have visibility of the feature of a remote hosts running Hyper-V. I wish it was here now - but better late than never.
Note that a VM's memory usage will not grow higher than its licenced limit, eg 4GB for client OS's and Standard Server, even if the Hyper-V SP1 Maximum RAM setting is deliberately or accidentally set to a higher value.
Correction to the above - limited to 4GB for 32-bit client OS's and 32-bit Standard Server
In VMware, there is a VM memory utilization metrics “Usage” as a performance counter provided by vCenter. It shows the memory utilization percentage of that VM.
For Hyper-V enabled Dynamic Memory, there is no memory utilization percentage or similar utilization field in Hyper-V API.
In order to calculate VM memory utilization percentage, we find out below Hyper-V WMI metrics can be used for calculation.
In Hyper-V WMI class, we can get below memory usage metrics from Msvm_SummaryInformation:
• MemoryAvailable (%)
• AvailableMemoryBuffer (%)
In Hyper-V WMI class, we can get below memory setting information from Msvm_MemorySettingData :
We define below formula to calculate the VM memory utilization. Please advise if that is correct or not. Or there is a much simple way to get the memory utilization percentage?
• Memory Demand = Memory Usage * (100 – Memory Available)/100
• Memory Capacity = MemoryUsage / (100- AvailableMemoryBuffer (%)*TargetMemoryBuffer (%) )
• Memory Consumed = Memory Demand amount that is met
o If Memory Demand < Memory Usage, then Consumed = Memory Demand
o If Memory Demand > Memory Usage, then Consumed = Memory Usage
• Memory Utilization = Memory Consumed / Memory Capacity * 100