VMware vSphere is definitely the most efficient Hypervisor in managing the Hardware resources of the host like CPU,Memory,Storage and Network . Understanding how these resources are efficiently managed by the Hypervisor would help Virtualization Administrators efficiently plan and manage their environment . This post is to give a basic idea on various components involved and the memory management techniques employed by VMware vSphere 4.1 . This blog is based on the Memory Resource Mangement in vSphere 4.1 white paper by VMware . Please note that each Virtual machine’s target memory allocation is dynamic and is dependent on current load on the host and memory allocated to the Virtual machine along with its associated parameters like Shares , Reservation and Limits .
Below are certain terminologies commonly used in VMware ESX whose understanding would help administrators understand these techniques better
- Host Physical memory - Amount of memory available on the host that is visible to the hypervisor.
- Guest Physical memory -Amount of memory that is visible to a guest OS running inside a VM.
- Consumed memory - This is the amount of host memory that has been allocated to a Virtual machine .
- Active memory - This is the amount of guest memory that is actively used by the applications and guest OS .
- Guest level paging -Transfer of Memory between the guest physical memory and guest swap . This is driven by the Guest OS.
- Hypervisor Swapping – Transfer of memory between guest physical memory and host swap . This is driven by the hypervisor.
- Guest virtual memory – Continuous virtual address space provided by the guest OS to its applications.
- Shares – Entitles a VM to a fraction of available host physical memory, based on a proportional-share allocation policy.
- Reservation – Guaranteed amount of minimum host physical memory the host reserves for a virtual machine .
- Limit – Maximum amount of host physical memory allocated for a virtual machine . Default is unlimited .
Memory Management and Reclamation techniques :
Ever wondered how ESX is able to over commit memory to the Virtual machines . By over commit I mean that the total sum of memory allocated to Virtual machines running on a host is greater than the available physical memory on the host . There are various memory reclamation techniques employed by ESX to achieve this namely ,
- TPS – Transparent Page sharing helps reclaim memory by removing redundant pages on the memory with identical content .
- Ballooning – Memory is reclaimed by artificially increasing the memory pressure inside a VM .
- Swapping : Hypervisor swapping help reclaim memory by swapping out guest memory onto swap created for every VM by hypervisor .
- Memory Compression : Reclaims memory by compressing pages that needs to be swapped out into Compression cache created for every VM on the host physical memory .
Transparent Page sharing :
When multiple Virtual machines running in a host are running same guest OS or at times same guest OS and Application , we would see a lot of identical content on the memory . Instead of storing redundant content it is enough if we store a single identical copy on the host physical memory and allow multiple VM’s to access this identical copy . This is the basic mechanism behind please note that this operation is invisible to the Virtual Machine and is performed at the hypervisor level and therefore there is no chance of leakage of sensitive information between Virtual machines .
Redundant pages are identified by their content and hashing algorithm is used to identify the redundant content from the guest physical memory . ESX scans the guest physical memory content at a specified rate for sharing opportunities . TPS is always ” ON ” and has the least overhead .
Ballooning :
Remember I had mentioned earlier in my post that a VM’s target memory allocation is dynamic and ESX allocates memory on demand to the VM based on the various parameters like shares , reservation , limits and host memory . Let’s take for example a VM has consumed 2 GB of RAM during its peak operations but is currently running with 1 GB active memory only . Say if there is a memory crunch on the host running this VM , Although the VM is currently using 1 GB of RAM only it will not release the additional 1 GB that it has freed since the VM is isolated and is not aware of host memory shortage on the hypervisor . Ballooning makes the Guest OS aware of the low memory status of the host and help reclaim memory .
Ballooning is achieved using Balloon drivers which are loaded into Guest OS running inside Virtual machines using VMware Tools . These drivers load as a pseudo-device driver and communicates with the hypervisor using a private channel . When the host in need of physical memory , the host will set a target balloon size for the balloon driver and in turn the balloon driver will inflate inside the Guest OS by allocating guest physical pages within the VM . What will happen is the Guest OS will page out the contents from the host physical memory . Once this is completed the balloon driver notifies the hypervisor which can now safely reclaim the freed memory . Please note that If the virtual machine has plenty of free guest physical memory ( as described in the example above ) , inflating the balloon will induce no paging and will not impact guest performance. Guest OS has to make the decision on what pages needs to be paged out and hypervisor has no call on this .
Ballooning is employed when these is memory pressure on the host . Ballooning will offload host memory pressure onto the VM as VM will now use less physical memory on the host and more physical memory inside the guest . Ballooning is employed as the first technique to reclaim memory whenever the host is under memory pressure . Please note that TPS is always enabled .
Swapping :
A Swap file is created for every VM on startup and is deleted when the VM is powered off . The size of the swap file is directly proportional to the amount of RAM that has been allocated to the VM . Swapping is invoked when memory reclaim from TPS and Ballooning is not sufficient enough to reduce host memory pressure . When swapping is invoked hypervisor will swap out guest physical memory to the swap file that was created on the VM startup . This will free up host physical memory and reduce the memory pressure on the host . This is a guaranteed technique to reclaim a specific amount of physical memory in the quickest time possible among the three techniques but then considering the overhead involved this is the last resort that ESX will use to reclaim memory . Please note that the swap file that has been created by the hypervisor for each and every VM is on the disk and Hypervisor swapping and further access of these pages by the VM will make the VM responsiveness slow .
Memory Compression :
Remember when hypervisor swapping is invoked pages are swapped out from guest physical memory to swap file created on the disk by hypervisor which will have latency . Memory Compression tries to compress the pages that needs to be swapped in a compression cache that is allocated per VM on the host physical memory itself . When these pages needs to be accessed b y the VM they will be uncompressed and presented which will be faster than a disk access in case of swapping . Please note that ESX will not proactively compress guest pages when host swapping is not required .
The per-VM compression cache is accounted for by the VM’s guest memory usage, which means ESX will not allocate additional host physical memory to store the compressed pages. If the compression cache is full, one compressed page must be replaced in order to make room for a new compressed page. An age-based replacement policy is used to choose the target page. The target page will be decompressed and swapped out. ESX will not swap out compressed pages.In ESX 4.1, the default maximum compression cache size is conservatively set to 10% of configured VM memory size .
There are four memory states that are maintained by the ESX Server
- High – 6% Threshold of Total Host Memory
- Soft – 4% Threshold of Total Host Memory
- Hard – 2% Threshold of Total Host Memory
- Low – 1% Threshold of Total Host Memory
Why are these States important ? What does these States denote ?
As mentioned earlier , TPS is enabled by default in the ESX Server . Host Memory free states in ESX Server determines when to invoke Ballooning and Swapping ( activates memory compression as well )
When the memory state is high , it indicates that the total amount of active memory used by the all VMs in a host is less than total host memory .This means that the host will not invoke ballooning or swapping as long as the state is high . This is valid only if limits are not set for the VM’s .
If host free memory drops towards the soft threshold, the hypervisor starts to reclaim memory using ballooning. Ballooning happens before free memory actually reaches the soft threshold because it takes time for the balloon driver to allocate and pin guest physical memory. Usually, the balloon driver is able to reclaim memory in a timely fashion so that the host free memory stays above the soft threshold.
If ballooning is not sufficient to reclaim memory or the host free memory drops towards the hard threshold, the hypervisor starts to use swapping in addition to using ballooning. During swapping, memory compression is activated as well. With host swapping and memory compression, the hypervisor should be able to quickly reclaim memory and bring the host memory state back to the soft state.
Note : In certain scenarios, host memory reclamation happens regardless of the current host free memory state. For example, even if host free memory is in the high state, memory reclamation is still mandatory when a virtual machine’s memory usage exceeds its specified memory limit. If this happens, the hypervisor will employ ballooning and, if necessary, swapping and memory compression to reclaim memory from the virtual machine until the virtual machine’s host memory usage falls back to its specified limit.
What are the benefits of these memory management techniques ?
- Better Utilization of available Memory
- Higher Consolidation Ratio
