A key concept that is very important to understand for the VMware administrator is how Shares, Limits and Reservations work. Why do we have these controls? Well to put it simply if we are over provisioning our ESXi host on memory and CPU we need a tool to make sure that the right machines get the correct amount of resources. An example could be: Make sure the important ERP system always gets 3000MHZ (Reservaton) and make sure that the Test system never gets more than 1000MHZ (Limit). This article will focus on the three concepts on Memory and CPU. You can also apply some of this on storage and network. But that is out of scope for this article.
A reservation is a guarantee on either memory or cpu for a virtual machine. You define the reservation in MB or MHZ. It works kind of different on the two compute resources.
On memory it is a guarantee for access to physical memory for the virtual machine. Remember, every virtual machine has a swap file. The swap file size is defined as (swap file = configured memory – memory reservation). When the virtual machine is running the vmkernel allocates memory to it when the VM requests it. The vmkernel will always try to map the memory to physical memory, but if the ESXi host is running low on memory the memory has to come from the swap file. This has a huge performance penalty. Lets take two examples:
- Example 1: You have a virtual machine configured with 2 GB memory and you configure a 1 GB reservation
- When the virtual machine powers on a 1 GB swap file (.vswp) is created on a datastore
- The 1 GB reservation guarantees that the VM will always a least get 1 GB of physical memory. If the ESXi host is running low the remaining 1 GB can come from the swap file on disk!
- Example 2: You have a virtual machine configured with 4 GB memory and you configure a 4 GB reservation
- When the virtual machine powers on a swap file with zero size is created
- The 4 GB reservation guarantees that the VM will get ALL its memory from physical memory and it will never do hypervisor swapping or ballooning.
On cpu the reservation is a guarantee for clock cycles. You define the reservation in MHZ. If you give a virtual machine a reservation it means the vmkernel CPU scheduler will give it at least that amount of resources. If a virtual machine is not using its resources the CPU cycles are not wasted on the physical host. Other machines can use it. What you do with CPU reservations is making sure that a VM will always get access to physical cpu in a committed environment.
A limit is a a limit 🙂 Defined on either Memory or CPU It is defined in MB or MHZ.
On memory the limit defines what is the maximum amount of physical memory the virtual machine can use. This is a very dangerous setting! If you set the limit lower than the configured memory for a VM it will cause swapping and balloon activity for the virtual machine.
- Example 1: You have a virtual machine configured with 4 GB memory and you configure a 1 GB limit
- What you have done is BAD, unless you want bad performance. The virtual machine guest operating system sees 4GB of memory (inside windows task manager for instance) but the ESXi is not allowed to give it more than 1 GB of physical memory. The virtual machine will probably request more that 1 GB for its application and when this happens ballooning and hypervisor swapping will start.
Limits on memory can have huge consequences. As a consultant I always look for virtual machines with a limit on memory, more often than you would imagine you find it configured on a few virtual machines!
On CPU the limit defines how much access a virtual cpu can get on a physical cpu (core). This is used to make sure a virtual machine is not using too much resources on a host. By limiting the vCPU you essentially also limits the performance of the virtual machine. Even though capacity is available on the ESXi host the limit will still be enforced.
Shares is a different approach to performance tuning in a virtual environment. Shares define how much access you get to a resource compared to something else. Every virtual machine has 1000 shares configured pr. vCPU as a default. So you are already using them! All virtual machine are equal from a hypervisor perspective unless you change the shares and tell it which machines are really more important. What is important to know about shares is that they only is considered in case of contention! If you have available capacity for all machines it does not help performance to increase the shares on some machines.
- Example 1: VM A has 1000 shares and VM B has 1000 shares and they are both competing for the same physical cpu core. In this case the vmkernel CPU scheduler will give each machine 1/2 or 50% access and they will have the same performance
- Example 2: VM A has 3000 shares and VM B has 1000 shares and they are both competing for the same physical cpu core the result would be that VM A gets 3/4 or 75% access and VM B gets 1/4 or 25% access.
- Example 3: VM A has 3000 shares and VM B has 1000 shares and they are not competing for the same physical cpu core. In this case both machines will get 100% access to physical CPU. Remember shares is only handled when we have contention!
Reservations, limits and shares are powerful resource controls in a virtual environment. Just make sure to think about how you implement it. Especially watch out for putting a memory limit on your virtual machine.
If you have any questions, please use the comments on this page.
Didier Pironet says
Limit is a feature left over by VMware developers to test memory management mechanisms such TPS, ballooning and so forth…
omid sabet says
hi, im just a little confuse here. you said when you configure limit less than configured memory memory balloon start, in other blogs you said that memory ballooning is a reclaim technic for host.
can you just explain more?
and thanks for great job you do here
If a VM is evacuated from one resource pool but later returns, will its reservations and limits settings remain? Or will they be erased?
Basavaraj.R Navalgund says
Very precise explanation.
Kailash Mishra says
Do shares have impact on cpu ready if host utilisation is 40% only
ballooning start when esxi host is under memory constraint. when you limit virtual machine memory and host has enough memory i think there will be no ballooning. is that true?
Correct.Balloning is to give memory back to the ESXi host.
CPU shares are only enforced when two vCPUs are fighting for the same resource. At 40% host the CPU ready time should be pretty low
Jose Hernandez says
Not quite. Even if the host has memory available, because a limit has been set in the VM, the host will begin the process of trying to reclaim memory on the VM which will use the balloon driver.
Search for ‘Impact of virtual machine memory and CPU resource limits’ for the kb on this.
Basavaraj Navalgund says
Thank you very much for this great explanation of reservation /limits and it helped to understand the core concepts.
can you configure a VM with 2GB RAM and 4GB RAM Reservation? guessing not….
No, you cannot do that. It makes no sense to reserve more than the VM can use.
Ahmed Fathy says
i have a question i dont seem to have an answer for, then i came across your solid explanation of things, so allow me to ask my question here:
For a single physical host, would i benefit from setting all VMs on this host to 100% cpu reservation, or its useless since i do the reservation on ALL vms on SAME host.
When you use reservation you cannot reserve more than you have. If you have 10.000 mhz you can only reserve 10.000 mhz to virtual machines. So with reservations you might limit yourself on the number of VM’s you can power on.
If all VM’s are on the same host today it will make little sense to reserve 100% to each one. ESXi will by itself provide pCPU resources to the individual virtual machines when needed.
A reservation is a “guarantee” that the VM will always gets it access to CPU resources. But it will potential limit the number of VM’s you can power on.
Thank you deeply for your fast response.
so from what i understand, and based on your example; assuming i have a host with 10mhz.
(dual sockets with 5mhz per) and i reserve all 10 mhz to one VM, as a result i “cannot” power on any other vms on this said host.
am i correct ?
That is correct, you can only power-on VM’s when the reservation can be satisfied. It makes no sense to be able to reserve more than you have.
You can very easily test this.
When you reserve CPU for a VM then you have to consider the speed of the physical processor. Lets say you have a Physical Quad Core 2,4 GHZ processor. Then you have 9600 MHZ total (you cannot reserve all since some is already reserved for the ESXi system services).
If you have a VM with a single vCPU and you want 100% reservation you would reserve 2400 MHZ – if the VM is configued with dual vCPU you would reserve 4800MHZ
hmm i see, much like memory reservation works.
Again, thank you for your assistance.