Tag Archives: counters

vCenter and vRealize counters – part 1

This blog post is adapted from my book, titled VMware vRealize Operations Performance and Capacity Management. It is published by Packt Publishing

vSphere 6 comes with many counters, many more than what a physical server provides. There are new counters that do not have a physical equivalent, such as memory ballooning, CPU latency, and vSphere replication. In addition, some counters have the same name as their physical world counterpart but behave differently in vSphere. Memory usage is a common one, resulting in confusion among system administrators. For those counters that are similar to their physical world counterparts, vSphere may use different units, such as milliseconds.

As a result, experienced IT administrators find it hard to master vSphere counters by building on their existing knowledge. Instead of trying to relate each counter to its physical equivalent, I find it useful to group them according to their purpose. Virtualization formalizes the relationship between the infrastructure team and application team. The infrastructure team changes from the system builder to service provider. The application team no longer owns the physical infrastructure.

The application team becomes a consumer of a shared service—the virtual platform. Depending on the Service Level Agreement (SLA), the application team can be served as if they have dedicated access to the infrastructure, or they can take a performance hit in exchange for a lower price. For SLAs where performance matters, the VM running in the cluster should not be impacted by any other VMs. The performance must be as good as if it is the only VM running in the ESXi.

Because there are two different counter users, there are two different purposes.

  • The application team (developers and the VM owner) only cares about their own VM.
  • The infrastructure team has to care about both the VM and infrastructure, especially when they need to show that the shared infrastructure is not a bottleneck.

One set of counters is to monitor the VM; the other set is to monitor the infrastructure. The following diagram shows the two different purposes and what we should check for each. By knowing what matters on each layer, we can better manage the virtual environment.

Book - chapter 3 - 01

At the VM layer, we care whether the VM is being served well by the platform. Other VMs are irrelevant from the VM owner’s point of view. A VM owner only wants to make sure his or her VM is not contending for a resource. So the key counter here is contention. Only when we are satisfied that there is no contention can we proceed to check whether the VM is sized correctly or not. Most people check for utilization first because that is what they are used to monitoring in the physical infrastructure. In a virtual environment, we should check for contention first.

At the infrastructure layer, we care whether it serves everyone well. Make sure that there is no contention for resource among all the VMs in the platform. Only when the infrastructure is clear from contention can we troubleshoot a particular VM. If the infrastructure is having a hard time serving majority of the VMs, there is no point troubleshooting a particular VM.

This two-layer concept is also implemented by vSphere in compute and storage architectures. For example, there are two distinct layers of memory in vSphere. There is the individual VM memory provided by the hypervisor and there is the physical memory at the host level. For an individual VM, we care whether the VM is getting enough memory. At the host level, we care whether the host has enough memory for everyone. Because of the difference in goals, we look for a different set of counters.

In the previous diagram, there are 2 numbers shown in a large font, indicating that there are 2 main steps in monitoring. Each step applies to each layer (the VM layer and infrastructure layer), so there are two numbers for each step:

  1. Step 1 is used for performance management. It is useful during troubleshooting or when checking whether we are meeting performance SLAs or not.
  2. Step 2 is used for capacity management. It is useful as part of long-term capacity planning. The time period for step 2 is typically 3 months, as we are checking for overall utilization and not a one off spike.

With the preceding concept in mind, we are ready to dive into more detail. Let’s cover compute, network, and storage in the next post.

Performance and Capacity Management in virtual datacenter

[30 Nov 2015: this was the reason why I wrote the 1st edition. While the 2nd edition has major changes, the high level goals of the 2nd edition remain the same]

Ever wonder what those counters mean in vCenter and vCenter Operations? For example, what does Memory Contention really mean, where does it get its value from, and what values you should expect in a healthy environment?

I was puzzled myself. I’ve been with VMware for 6+ years, and I do not see them being documented. Scott Drummonds did document some of them at communities.vmware.com when he was doing performance. But I have not seen a complete one, especially one written from VMware Admin’s view point (as opposed from a product view point). I want to explain from the view point of the person whose hands are on the keyboard when the VMware platform is not performing as expected.

I’m writing a book to address it. As a field person, I work closely with customers. I see first hand that there are related issue which makes performance and capacity management difficult in virtual world. The gaps explain why customers struggle with virtualisation. Because there are several gaps, I address them “top down“, starting from high level then move to technical matters. There are >100 pages on counters, so this is going to get deep 🙂

Chapter 1 covers the big picture. I aim to correct some architecture and operations mistakes that customers make. It is very rare for customers to master the virtual world, both architecturally and operationally. Part of the reason is the software-Defined Data Center (SDDC) is not yet matured. The biggest reason, however, is the lack of deep understanding of what exactly virtualisation means to IT. As we all know, small leaks sink the ship, so I’m going to expose the details.

Chapter 2 continues the theme, but I dive into Capacity Management. I’m hoping to share how you do capacity management. Once you know what it takes, you can use a tool to automate it.

Chapter 3 dives into the metrics and how CPU, RAM, Disk and Network change once you add a layer called hypervisor. You need to know this, as there are behaviour that completely change.

Chapter 4 – 7 document the counters in vCenter and vC Ops. One chapter for CPU, RAM, Network and Disk. The book goes beyond documenting the metrics. It also applies them so customers can operationalise their virtual platform.

Chapter 8 provides examples of dashboards. The book was practically born in the field, sitting down with customers in creating all those dashboards and performing troubleshooting.

The book is with Packt Publishing. There are many reasons why I chose to work with them. The plan is to have the book out at the same time with the GA of vRealize Operations 6.