Tag Archives: ESXi

Your ESXi Host needs more RAM. Really?

This blog post is adapted from my book, titled VMware vRealize Operations Performance and Capacity Management. It is published by Packt Publishing. You can buy it at Amazon or Packt.

I mentioned earlier that vRealize Operations does not simply regurgitate what vCenter has. Some of the vSphere-specific characteristics are not properly understood by traditional management tools. Partial understanding can lead to misunderstanding. vRealize Operations starts by fully understanding the unique behavior of vSphere, then simplifying it by consolidating and standardizing the counters. For example, vRealize Operations creates derived counters such as Contention and Workload, then applies them to CPU, RAM, disk, and network.

Let’s take a look at one example of how partial information can be misleading in a troubleshooting scenario. It is common for customers to invest in an ESXi host with plenty of RAM. I’ve seen hosts with 256 to 512 GB of RAM. One reason behind this is the way vCenter displays information. In the following screenshot, vCenter is giving me an alert. The host is running on high memory utilization. I’m not showing the other host, but you can see that it has a warning, as it is high too. The screenshots are all from vCenter 5.0 and vCenter Operations 5.7, but the behavior is still the same in vCenter 5.5 Update 2 and vRealize Operations 6.0.

I’m using vSphere 5.0 and vCenter Operations 5.x to show the screenshots as I want to provide an example of the point I stated in Chapter 1 of the book, which is the rapid change of vCloud Suite and VMware SDDC products. I have verified that the behaviour is still the same in vSphere 5.5 U2.


The first step is to check if someone has modified the alarm by reducing the threshold. The next screenshot shows that utilization above 95% will trigger an alert, while utilization above 90% will trigger a warning. The threshold has to be breached by at least 5 minutes. The alarm is set to a suitably high configuration, so we will assume the alert is genuinely indicating a high utilization on the host.


Let’s verify the memory utilization. I’m checking both the hosts as there are two of them in the cluster. Both are indeed high. The utilization for vmsgesxi006 has gone down in the time taken to review the Alarm Settings tab and move to this view, so both hosts are now in the Warning status.


Now we will look at the vmsgesxi006 specification. From the following screenshot, we can see it has 32 GB of physical RAM, and RAM usage is 30747 MB. It is at 93.8%utilization.


Since all the numbers shown in the preceding screenshot are refreshed within minutes, we need to check with a longer timeline to make sure this is not a one-time spike. So let’s check for the last 24 hours. The next screenshot shows that the utilization was indeed consistently high. For the entire 24-hour period, it has consistently been above 92.5%, and it hits 95% several times. So this ESXi host was indeed in need of more RAM.


Deciding whether to add more RAM is complex; there are many factors to be considered. There will be downtime on the host, and you need to do it for every host in the cluster since you need to maintain a consistent build cluster-wide. There will be lots of vMotion as each host needs to be shutdown. Because the ESXi is highly utilized, I should increase the RAM significantly so that I can support more VMs or larger VMs. Buying bigger DIMMs may mean throwing away the existing DIMMs, as there are rules restricting the mixing of DIMMs. Mixing DIMMs also increases management complexity. The new DIMM may require a BIOS update, which may trigger a change request. Alternatively, the large DIMM may not be compatible with the existing host, in which case I have to buy a new box. So a RAM upgrade may trigger a host upgrade, which is a larger project.

Before jumping in to a procurement cycle to buy more RAM, let’s double-check our findings. It is important to ask what is the host used for? and who is using it?.

In this example scenario, we examined a lab environment, the VMware ASEAN lab. Let’s check out the memory utilization again, this time with the context in mind. The preceding graph shows high memory utilization over a 24-hour period, yet no one was using the lab in the early hours of the morning! I am aware of this as I am the lab administrator in the past 6+ years!

We will now turn to vCenter Operations for an alternative view. The following screenshot from vCenter Operations 5 tells a different story. CPU, RAM, disk, and network are all in the healthy range. Specifically for RAM, it has 97% utilization but 32% demand. Note that the Memory chart is divided into 2 parts. The upper one is at the ESXi level, while the lower one shows individual VMs in that host.

The upper part is in turn split into 2. The green rectangle (Demand) sits on top of a grey rectangle (Usage). The green rectangle shows a healthy figure at around 10 GB. The grey rectangle is much longer, almost filling the entire area.

The lower part shows the hypervisor and the VMs’ memory utilization. Each little green box represents 1 VM.

On the bottom left, note the KEY METRICS section. vCenter Operations 5 shows that Memory | Contention is 0%. This means none of the VMs running on the host is contending for memory. They are all being served well!


I shared earlier that the behavior remains the same in vCenter 5.5. So, let’s take a look at how memory utilization is shown in vCenter 5.5.

The next screenshot shows the counters provided by vCenter 5.5. This is from a different ESXi host, as I want to provide you with a second example. It is not related to the 1st example at all. This host has 48 GB of RAM. About 26 GB has been mapped to VM or VMkernel, which is shown by the Consumed counter (the highest line in the chart; notice that the value is almost constant). The Usage counter shows 52% because it takes from Consumed. The active memory is a lot lower, as you can see from the line at the bottom. Notice that the line is not a simple straight line, as the value goes up and down. This proves that the Usage counter is actually the Consumed counter.

vSphere 5.5 ESXi memory counters

Notice that the ballooning is 0, so there is no memory pressure for this host. Can you explain why? It is because Consumed is below the threshold.

At this point, some readers might wonder whether that’s a bug in vCenter. No, it is not. ESXi is simply taking full advantage of the extra RAM. This is a similar behaviour by Windows 7 and Windows 2008.

There are situations in which you want to use the consumed memory and not the active memory. In fact, some applications may not run properly if you use active memory. I will cover this when we discuss memory counters in further chapters. Also, technically, it is not a bug as the data it gives is correct. It is just that additional data will give a more complete picture since we are at the ESXi level and not at the VM level. vRealize Operations distinguishes between the active memory and consumed memory and provides both types of data. vCenter uses the Consumed counter for utilization for the ESXi host.

As you will see later in this book, vCenter uses the Active counter for utilization for VM. So the Usage counter has a different formula in vCenter depending upon the object. This makes sense as they are at different levels. In my book I explain these 2 levels.

vRealize Operations uses the Active counter for utilization. Just because a physical DIMM on the motherboard is mapped to a virtual DIMM in the VM, it does not mean it is used (read or write). You can use that DIMM for other VMs and you will not incur (for practical purposes) performance degradation. It is common for Microsoft Windows to initialize pages upon boot with zeroes, but never use them subsequently.

Having said all the above, none of the counters & information above should be used in isolation. You should always check the VMs. It’s a common mistake to look at ESXi RAM counters (be it Consumed, Active, Usage, or Workload) and conclude there is RAM performance or RAM capacity issue. You should always check the VMs first. Are they having Memory Contention? If not, there is no RAM performance issue. What about Capacity? It’s the same principle. Take into account performance first, before you look at utilization. See this series of articles on what super metric you need.

For further information on this topic, I would recommend reviewing Kit Colbert’s presentation on Memory in vSphere at VMworld, 2012. The content is still relevant for vSphere 5.x. The title is Understanding Virtualized Memory Performance Management and the session ID is INF-VSP1729. If the link has changed, here is the link to the full list of VMworld 2012 sessions.

Not all performance management tools understand this vCenter-specific characteristic. They would have given you a recommendation to buy more RAM.

Now that you know your ESXi does not need so much RAM, you can consider a single socket ESXi.

[20 March 2016: added 3rd example]

User Jengl at VMware communities posted a question here. He wanted to know why, when “we patched the half of our cluster we had high consumed/usage of memory and the vCenter started to initiate balloon, compress and finally swap on the ESXi Hosts. I was not expecting that, because active memory was only a small percentage of consumed.

He shared his screenshot, which I copied here so I can explain it.


In the above screenshot, it looks like ESX Host (and not cluster or VM). We could see around 10:50 am, there was a spike. The Memory Usage counter hit 98.92%. That’s high, as that’s an average of 5 minutes. It probably touched 100% within that 5 minutes. Memory Workload, which is based on Active, was low at 19%. This is possible. The VMs were not actively using the RAM. Contention rose to a 1.82%, which indicates some VMs were contending for RAM.

The whole situation went back to normal. Contention disappeared pretty quickly, and Memory Usage went down. It’s hard to see fro the chart, but it looks like it dropped below 95%.

Based on the above, I’d expect balloon to have kicked in at 10:50 am too. It would then taper off. Because the VMs were not actively using the RAM, I’d not expect the value to drop to 0.

Guess what? The screenshot below confirmed that.


I hope that’s clear.

SCP copy failed between ESX hosts!!

Learned a couple of tricks around SCP while using it lately.

There are many posts available online that will show the method to copy files between ESXi hosts. There is one particular post which I liked is from Hersey Cartwright. You can read it here.

However, it didn’t work in my case. I was trying to move a VM across two different vCenters. After issuing the command on SSH from the source host there was no error, no prompt….nothing.

Below is the command that I have used.

# scp -r <VMName> root@<Destination Host>:/vmfs/volumes/<Datastore on Destination host>

So, here’s the trick. We all know that we need to start SSH daemon for SCP to work. Little did I know that there is need to start SSH client (Outgoing Port) under “Firewall Properties” on source ESXi Host.

Screen Shot 2015-02-06 at 2.20.49 pm

Once the changes are applied, re-run the same command and this time you will be prompted for password for destination host. If the authentication is successful, the file copy will start.

I hope that vMotion in vSphere 6 is going to take care of this in future, but we will have to wait and watch the real life use around it.

The rise of single-socket ESXi Host

When I first started doing VMware back in mid 2008, it was common to see 4-socket ESXi host. A cluster of 8 nodes means the cluster will have 32 sockets. This means 32 licenses of vSphere. On top of that, customers also have to pay for Guest OS. Some customers had to pay both RedHat and Windows.

With each passing year, Intel delivered more and more cores. I have customers who have excess vSphere license as they went from dual-core to 12-core over several years.

Fast forward to the current time. Intel launched an 18-core Xeon E5-2699 V3 in Q4 2014, followed by Xeon E6-2699 V4 launched in Q1 2016, and then Xeon Platinum 8176 in July 2017. This sports 28 cores! AMD has also joined in with Epyc.

The VMmark result shows near linear performance compared with older Xeon. Some of my customers have managed to reduce the number of ESXi Hosts. This is a good news to them, as that means less:

  • power consumption
  • UPS facility
  • data center space
  • air-cond facility
  • smaller VMware environment (for large customer, this makes management easier)
  • fewer vSphere licence (which means they can use the maintenance budget to get NSX, vSAN and vRealize)
  • less Windows Datacenter licence as it gives unlimited VM
    • Note: this does not apply to a single socket. See below.
  • less RHEL license as it gives unlimited VM
  • less software license that charge per physical socket. For example, if you run Oracle softwares or Microsoft SQL Server, the savings will be more than the infrastructure saving.

Going forward, I see customers with large ESXi farm to further increase their consolidation ratio in the next 1-2 years. I see that 20:1 is becoming common. This means

  • 15:1 for Tier 1 workload
  • 30:1 for Tier 2 workload
  • 60:1 for Tier 3 workload (double of Tier 2, as price is also half)

On the other scale, I see customers with very small number of VM to go down to 1 socket ESXi. This actually opens up a possibility for use cases that vSAN, NSX or vSphere could not address due to cost. Remote branches or ROBO is such a use case. In this use case, a 4-node cluster for vSAN may not make financial sense. That means 8 license. By going single-socket, the cost is reduced by 50%.

Thanks to Patrick Terlisten (@PTerlisten) and Flemming Riis (@FlemmingRiis) who corrected me on Twitter that Windows Datacenter Edition comes with 2 physical socket entitlement. It cannot be split into 2 separate physical servers. I found that a document from Microsoft titled “Volume Licensing reference guide Windows Server 2012 R2”, dated Nov 2013, stated it clearly on page 10:

Can I split my Windows Server 2012 R2 license across multiple servers?
No. Each license can be assigned only to a single physical server.

The extra core also supports the converged architecture. Storage and Network can now be run in software. We can use the extra core for services such as:

  • vSAN
  • vSphere Replication
  • vSphere Data Protection
  • NSX distributed router
  • NSX Edge
  • Trend Micro anti virus
  • F5 load balancer
  • Palo Alto Network firewall
  • etc

With a single-socket, the form factor has to be 1RU max. 2RU will be considered too space-consuming. In some cases such as VxRail, Supermicro and Nutanix, the 2RU form factor actually hosts 4 nodes, making each node 0.5 RU so to speak.

In the 1RU form factor, you do not have to compromise storage. For example, the Dell PowerEdge r630 provides 24 SSD, giving you 23 TB raw capacity. It has 24 x 1.8” SSD – up to 23 TB via 0.96 TB hot-plug SATA SSD.

You also do not need to compromise on RAM. Review this post to see that it’s a common mistake to have ESXi with excess RAM.

We know that Distributed Storage works much better on 10Gb networking than 1Gb networking. To get a pair of 24-ports 10Gb switch can be cost prohibitive. The good thing is there are vendors who supply 12-port switches, such as XSNetGear and ZyXEL.

I’d like to end this post by getting it validated by practitioners, folks who actually run technology like VSAN in production to see if this single-socket idea makes sense. I discussed this with Neil Cresswell, CEO of Indonesian Cloud, and this is what he has to say:

“With the rapidly increasing performance of CPUs, once again the discussion of scale up (fewer big servers) or scale out (more small servers) comes back into play. I remember a client a few years ago that bought an IBM 8 Socket Server and were running 100 VMs on it; at that time I thought they were crazy; why do that vs having 2x 4 socket servers with 50 VMs each! Well, that same argument is now true for 2 socket vs single socket. You can have a dual socket server running 50VMs or 2x single socket servers each running 25 VMs.
As a big believer in risk mitigation, I for one would always prefer to have more servers to help spread my risk of failure (smaller failure domain).”

What’s your take? Do you see customers (or your own infrastructure) adopting single-socket ESXi host? Do you see distributed storage at ROBO becomes visible with this smaller config?

I’m keen to hear your thought.