Tag Archives: vRealize Operations 6.4

Multi-tier Application Monitoring

This post is part of Operationalize Your World post. Do read it first to get the context.

I covered a single-tier application in this post. Read that first, as this blog builds upon that.

Done?

Great! A lot of you have shared that you want a multi-tier applications. One customer has a mission critical application that spans 5 tiers and 68 VMs. The dashboard I shared earlier does not scale to that level, as you certainly don’t want to check 68 VM one by one!

Application

To make sure we are on the same page, here is an example of multi-tier application:

A multi-tier application can suffer from either horizontal or vertical problem:

  • By horizontal, I mean a tier has problem. When the web tier is slow, it can slow down the entire application. The speed of a convoy is determined by the slowest car. We will use the following formula to determine the application performance:
    Application Performance = Minimum (Tier Performance)
  • By vertical, I mean something that cut across tier. Storage, for example. If the slowness is caused by something common, there is no need to troubleshoot individual VM, as they are simply victim.

That means we need to check both angles when an application had performance problem:

  • Which tier had the problem? Since when? How bad? What was the problem?
  • What infra problem did the app had? Storage, Network, CPU, RAM?

The above check makes a good starting point in your analysis. Don’t zoom into a particular VM until you know the overall picture. No point fire fighting the kitchen if the whole house is on fire.

Application Tier

The health of a tier is the average health of its member. This is because a tier scales out. We are not taking the minimum value. This is not a convoy.

Hold on!” you might say. Since it is scale out, App Team has catered for this. If they only need 3 web servers, they will deploy 4 or even 5. So both performance and availability are not affected. The tier performance has to take into account this extra node, not simply doing an average.

This logic sounds reasonable. But is it correct?

It is not actually. Because this is not about Availability. This is about Performance. All web servers are still up, but if node no 4 is slower, user experience will be affected.

My fellow blogger Luciano Gomes advise that some Load Balancer can detect the performance of a node. This is good, as simply counting the number of session is not a complete measurement. The node measurement is based on this formula. It takes into account how the IaaS platform is serving the Node. So it’s looking beyond the Guest OS. This is because performance cannot be measured from within the Guest OS only. Review this discussion between David Davis and Sunny Dua.

This is why we are doing an average. You want to be informed if there is degradation, since this is performance, not availability.

The health of a VM is simple. We leverage the work we’ve done for a single-tier application here.

Dashboard 

I found that doing a logical design of dashboard actually saves time. Follow best practices to help you.

Here is the logical design of the dashboard. Notice it has 3 levels: App, Tier, VM.

Here is what it looks like

Click on the image below to see explanation of each section:

Hope you find it useful. As usual, you do not have to build this from scratch. This is part of Operationalize Your World, which give you 50+ dashboards.

What’s new with vRealize Operations 6.4

Apology that this is coming late. Please read the following first, as I will start from where they end:

I’d focus on the new dashboards, since I was the one designing them. If the dashboards are not meeting your requirements, you now know exactly where to complain 😉 The dashboards were reviewed extensively by Product Managers (Monica Sharma, Ronit Halachmi Bekel) and Sunny Dua.

The dashboards in 6.4 is a subset of the dashboards in Operationalize Your World program. Around 20% made it. They are also simplified. The reason is we wanted the dashboards to pass The 5-second Test, and be applicable to SMB segment. We also wanted to have more feedback from real life environment, before bringing additional & more advanced dashboards. So do let me know at e1@vmware.com.

The following screenshot shows the 2 sets when you are running 6.4. You end up with both sets. They can co-exist, and the cost is some metrics are duplicated. As part of porting the dashboards to 6.4, we converted the super metrics into regular metrics so it’s simpler for you.

2016-12-10

Do we remove the old dashboards in 6.3? Nope, we did not. We simply moved them. Can you guess where they are on the screenshot above?

Yes, we moved them under “Other” folder. In future, we might deprecate them as we enhance the UI and dashboards.

You will notice we have grouped the dashboards into Infrastructure and VM. This is in-line with the Dining Area and Kitchen shared in Operationalize Your World. We wanted to drive your attention that you should ensure the VMs are served well, before you look at the kitchen.

We’ve placed some dashboards outside the folder for your convenience. We’ve also created a Read Me dashboard. We call it Getting Started. It explains the new dashboards.

getting-started

Technically, the dashboard has only 1 Text Widget. The adventurous among you will ask me if you can clone and tailor it for your company. The answer is yes. No, it is not supported. All the texts and images are in this directory:

ContentPack/VMWARE/conf/pages/getting_started/index.html?locale=%locale%

Operations Overview dashboard

We designed this dashboard to answer a few frequently asked questions on your day to day operations.

  • What have we got? If this number change, or not what you expect, you want to probe why.
  • What’s the Health of my environment? Environment can vary in size, so we group them by vSphere Data Center. The dashboard lists all your data centers. Select one, and you can see its uptime and alerts. You expect the uptime to be 100% and the Alerts to be below your normal operations.
  • Just because your vSphere is healthy does not mean the VMs are being served well. This is where the Top-N comes in. As this dashboard is your daily dashboard, you should expect the number to be within your expectation.

ops-overview

Cluster Performance dashboard

If the VMs are not being served well, you need to investigate why. This is where the following dashboard comes into play. It lets you see which clusters are not performing well. The heatmap shows the cluster by alert. Start with the reddest cluster.

Select the cluster you want to probe. Its performance counters will be automatically shown. You can see if it’s serving its VMs well. We are using line chart, so you can see the past and check if there is any strange spike.

cluster-performance

Heavy Hitter VMs dashboard

One possible cause of performance is you have Villain VM. Your vSphere environment is a shared environment. It can take as little as 1-2 VM to create performance problem in a cluster with 500 VMs.

This dashboard answers if there is any abnormal spike generated by any VM. It tracks both Storage and Network. From the following example, we can easily see there are both excessive storage and very high network throughput. They happened on different time and were caused by different VMs. The dashboard quickly shows the 2 villain VMs. You can see their workload is >10x to the second highest VM.

heavy-hitter-vms

In the Operationalize Your World, we enhance this dashboard by adding details, and split it into 2 (Storage and Network). This is to facilitate collaboration with your peers.

Datastore Performance dashboard

Cluster covers compute. What about storage? The Datastore Performance dashboard lets you see the performance of all your datastores. You can use the view to select a datastore. Its performance charts will be automatically shown. From the line chart, you could see if the datastore was having difficulty serving its VMs.

You can drill down to see each VM in the datastore. Select a VM, and its IOPS and Latency will be automatically plotted.

datastore-performance

VM Performance Troubleshooting dashboard

There are 2 spectrum of performance problem:

  • Whole house on fire
  • A small number of VMs were hit

The first few dashboards help you answer the first use case. This dashboard helps you look at a single VM. It’s a big dashboard, so I’ve added visual sections. It has 3 sections.

  • Section 1
    • This is where you select the VM. You can search, filter or simply browse.
    • The selected VM key properties, alerts and how it fits into the larger environment are automatically shown. If a VM is part of Resource Pool, check if the Resource Pool is limiting it.
  • Section 2
    • We display both the VM KPI and the IaaS KPI.
    • IaaS KPI is the 4 key metrics that shows how the IaaS serves this VM. If this is high, there is a good chance your IaaS capacity is full. It is struggle to serve all its VMs.
  • Section 3
    • You are verifying if the underlying IaaS was able to serve the VM.
    • Can you guess why we show Cluster instead of ESXi?
    • Hint: the performance problem may happened in the past.

vm-troubleshoot

In the Operationalize Your World, we expanded this dashboard into a set of dashboard.

VM Usage dashboard

A common request from VM Owner is to get his VM utilization and property. This is what this simple dashboard is for. You simply select the VM, and the key information is automatically shown. We are using line chart again, as the data that the VM Owner wants to know can be in the past.

If you need something more advanced, with self service, review the tenant dashboard here.

vm-usage

Capacity Dashboards

The twin brother of performance is capacity. While they are different, they are closely related. We’ve provided 2 dashboards to get you going:

  • Cluster capacity
  • Datastore capacity

The Cluster capacity lets you see a cluster utilization in 3 areas: CPU, RAM and Disk.

The model we use here is based on utilization. It does not take into account Availability Policy and Performance SLA. It is also based on Demand model, which is not suitable if you are doing Tier 1 cluster.

capacity-overview

The datastore capacity lets you see quickly which datastore is running out of space, and which datastores are hardly used. It uses the red color to show low capacity, and dark grey to show wastage. What you want to see is balanced usage across all datastores.

datastore-capacity

Configuration Dashboards

The last set of dashboards cover Configuration. We focus on configuration that need attention, rather than simply listing all configuration. Take the VM configuration dashboard, shown below. It highlights is you have lrge VMs, how large they are, and how many for each size.

You can also customize the filter. Simply edit the view widget.

It also highlight configuration that you need to watch. A VM with > vNIC should get your attention that it can bridge your network.

vm-config

We apply the same principle to the ESXi Configuration dashboard. For example, it shows the BIOS version. You want to keep the version consistent and minimal.

esxi-conf

The Cluster Configuration highlight inconsistent config among members ESXi in the cluster.

cluster-conf

The Network configuration lets your peers in the Network team to quickly understand the virtual network. It lists all the distributed virtual switch. Once you select one, it automatically lists all the port groups and ESXi in that switch. It also lists all the VMs. You can control and customise all these lists.

network-config

Hope you found them useful. We have intentionally kept them simple in 6.4. If you are running 6.3 or later, and you need a more advanced dashboards, download from here.