Understanding VMware Reservations Limits and Shares

Introduction

A key concept that is very important to understand for the VMware administrator is how Shares, Limits and Reservations work. Why do we have these controls? Well to put it simply if we are over provisioning our ESXi host on memory and CPU we need a tool to make sure that the right machines get the correct amount of resources. An example could be: Make sure the important ERP system always gets 3000MHZ (Reservaton) and make sure that the Test system never gets more than 1000MHZ (Limit). This article will focus on the three concepts on Memory and CPU. You can also apply some of this on storage and network. But that is out of scope for this article.

Rerservations

A reservation is a guarantee on either memory or cpu for a virtual machine. You define the reservation in MB or MHZ. It works kind of different on the two compute resources.

On memory it is a guarantee for access to physical memory for the virtual machine. Remember, every virtual machine has a swap file. The swap file size is defined as (swap file = configured memory – memory reservation). When the virtual machine is running the vmkernel allocates memory to it when the VM requests it. The vmkernel will always try to map the memory to physical memory, but if the ESXi host is running low on memory the memory has to come from the swap file. This has a huge performance penalty. Lets take two examples:

• Example 1: You have a virtual machine configured with 2 GB memory and you configure a 1 GB reservation
  • When the virtual machine powers on a 1 GB swap file (.vswp) is created on a datastore
  • The 1 GB reservation guarantees that the VM will always a  least get 1 GB of physical memory. If the ESXi host is running low the remaining 1 GB can come from the swap file on disk!
• Example 2: You have a virtual machine configured with 4 GB memory and you configure a 4 GB reservation
  • When the virtual machine powers on a swap file with zero size is created
  • The 4 GB reservation guarantees that the VM will get ALL its memory from physical memory and it will never do hypervisor swapping or ballooning.

On cpu the reservation is a guarantee for clock cycles. You define the reservation in MHZ. If you give a virtual machine a reservation it means the vmkernel CPU scheduler will give it at least that amount of resources. If a virtual machine is not using its resources the CPU cycles are not wasted on the physical host. Other machines can use it. What you do with CPU reservations is making sure that a VM will always get access to physical cpu in a committed environment. 

Limits

A limit is a a limit 🙂 Defined on either Memory or CPU It is defined in MB or MHZ.

On memory the limit defines what is the maximum amount of physical memory the virtual machine can use. This is a very dangerous setting! If you set the limit lower than the configured memory for a VM it will cause swapping and balloon activity for the virtual machine.

• Example 1: You have a virtual machine configured with 4 GB memory and you configure a 1 GB limit
  • What you have done is BAD, unless you want bad performance. The virtual machine guest operating system sees 4GB of memory (inside windows task manager for instance) but the ESXi is not allowed to give it more than 1 GB of physical memory. The virtual machine will probably request more that 1 GB for its application and when this happens ballooning and hypervisor swapping will start.

Limits on memory can have huge consequences. As a consultant I always look for virtual machines with a limit on memory, more often than you would imagine you find it configured on a few virtual machines! 

On CPU the limit defines how much access a virtual cpu can get on a physical cpu (core). This is used to make sure a virtual machine is not using too much resources on a host. By limiting the vCPU you essentially also limits the performance of the virtual machine. Even though capacity is available on the ESXi host the limit will still be enforced.

Shares

Shares is a different approach to performance tuning in a virtual environment. Shares define how much access you get to a resource compared to something else. Every virtual machine has 1000 shares configured pr. vCPU as a default. So you are already using them! All virtual machine are equal from a hypervisor perspective unless you change the shares and tell it which machines are really more important. What is important to know about shares is that they only is considered in case of contention! If you have available capacity for all machines it does not help performance to increase the shares on some machines.

• Example 1: VM A has 1000 shares and VM B has 1000 shares and they are both competing for the same physical cpu core. In this case the vmkernel CPU scheduler will give each machine 1/2 or 50% access and they will have the same performance
• Example 2: VM A has 3000 shares and VM B has 1000 shares and they are both competing for the same physical cpu core the result would be that VM A gets 3/4 or 75% access and VM B gets 1/4 or 25% access.
• Example 3: VM A has 3000 shares and VM B has 1000 shares and they are not competing for the same physical cpu core. In this case both machines will get 100% access to physical CPU. Remember shares is only handled when we have contention!

Summary

Reservations, limits and shares are powerful resource controls in a virtual environment. Just make sure to think about how you implement it. Especially watch out for putting a memory limit on your virtual machine.

If you have any questions, please use the comments on this page.