Tag Archives: Storage

Datastore Capacity Management

This post is part of Operationalize Your World post. Do read it first to get the context.

This is the 2nd installment of Storage Capacity Management. The previous post covers the overall storage capacity management, where you can see the big picture and know which datastores are low in capacity. This post drills further and lets you analyze a specific datastore.

Datastore capacity is driven by 2 factors:

  • Performance: If the datastore is unable to serve its existing VMs, are you going to add more VM? You are right, the datastore is full, regardless of how space it has left.
  • Utilization: How much capacity is left? Thin provisioning makes this challenging.

This is what the dashboard looks like.

You start by selecting a datastore you want to check. This step is actually optional, as you would have come from the overall dashboard.

When you select a datastore, its Performance and Utilization are automatically shown.

  • Performance
    • Both actual and SLA are shown.
    • You just need to ensure that actual does not breach SLA.
  • Utilization
    • This shows the total capacity, the provisioned capacity (configured to the VM), and what’s actually used (thin provisioned).
    • You want to be careful with thin provisioning, as the VM can consumed the space as it’s already allocated to it. The line chart has 30-day projection to help you plan.

The 2 line charts is all you need. It is simple enough, yet detailed enough. It gives you room to make the judgement call. You can decide to ignore the spike because you knew it was a special event.

If you want to analyse, you can see the individual VMs. The heatmap shows the VMs distribution. You can see if there are large VMs, because they are bigger. You can see if any VM is running out of capacity, or any VM is wasting the allocated capacity.

The heatmap configuration below shows how it’s done.

You can also check if there are VMs that you can delete. Reclamation will give you extra space. The heatmap has a filter for powered off VMs, so only powered off VMs are shown.

From there, you can drill further to check that the VM has indeed met your Powered Off definition. It’s showing the VM powered off time (%) in the past 30 days. I’ve set the threshold to be 99%. Green means the VM is at least 99% powered off in the past 30 days.

Logic

I hope you agree by now that datastore performance is measured on how well it serves its VMs. We can track this by plotting a line chart showing the maximum storage latency experienced by any VM in the datastore. This maximum number has to be lower than the SLA you promise at all times.

For Utilization, we will plot a line chart showing the disk capacity left in the datastore cluster.

You should be using Datastore Cluster. Other than the benefits that you get from using it, it also makes capacity management easier.

  • You need not manually exclude local datastore.
  • You need not manually group the shared datastores, which can be complex if you have multiple clusters.

With vSAN, you only have 1 datastore per cluster and need not exclude local datastores manually. This means it’s even simpler in vSAN.

Include buffer for snapshot. This can be 20%, depending on your environment. This is why I’m not a fan of many small datastores, as you have pockets of unusable capacity. This does not have to be hardcoded in your super metric, but you have to be mentally aware of it.

Super Metrics

The screenshot below shows the super metric formula to get the Maximum latency of all the VMs in the cluster. I’ve chosen at Virtual Disk level, so it does not matter whether it is VMFS, VMFS, NFS or VSAN.

super metric - vDisk

You can copy paste the formula below:

Max ( ${adapterkind=VMWARE, resourcekind=VirtualMachine, attribute=virtualDisk|totalLatency, depth=2 } )

The screenshot below shows the super metric formula to get the total number of disk capacity left in the cluster. This is based on Thin Provisioning consumption.

You can copy paste the formula below:

sum( ${adapterkind=VMWARE, resourcekind=Datastore, attribute=capacity|available_space, depth=1} )

For Thick Provision, use the following super metric:

super metric - Disk - space left in datastore cluster - thick

You can copy paste the formula below:

Sum
(
${adapterkind=VMWARE, resourcekind=Datastore, attribute=capacity|total_capacity, depth=1}
) –
Sum
(
${adapterkind=VMWARE, resourcekind=Datastore, attribute=capacity|consumer_provisioned, depth=1}
)

Hope you find it useful. Just in case you’re not aware, you don’t have to implement all these manually. You can import this dashboard, together with 50+ others, from this set.

Storage Capacity Management

This post is part of Operationalize Your World post. Do read it first to get the context.

We showed the dashboards in vSphere visibility for Storage Team post. Let’s now show how they are implemented.

BTW, you don't have to know how they are implemented.
You can simply imports all the ~50 dashboards by following the steps here.

For convenience, here is what the dashboard looks like. From this overall dashboard, you can drill down into datastore-level capacity planning.

The Summary is just a scoreboard widget. Here is the config.

It’s using 3 groups, which you have to create as part of the import process. The 3 groups are:

  • Datastores (Shared)
  • Powered On VMs
  • Powered Off VMs

You can customize the threshold. Do note that it applies to all service tiers.

The bar chart is just a view widget. Since I already have a group for Shared Datastores, it’s a matter of selecting them. It’s faster than selecting the World object, then filter the datastore. In the following screenshot, that’s the name of the view.

The 3 heatmaps are documented below.

I showed you the Datastore Cluster first as you should take advantage of Datastore Cluster. It makes operations easier compared with datastore.

The datastore heatmap again leverages the group. This filters out local datastores, which is what you want.

The view widget also leverages the group, for the same reason I shared above.

For the heatmap “Are the VMs using the storage given to them?“, I’m only showing the powered on VMs.

Hope that helps for overall storage capacity management. For capacity management at the datastore level, review this.

vSphere visibility for Storage Team

This post is part of Operationalize Your World post. Do read it first to get the context.

Ask any Storage Team and Platform Team whether the collaboration between them can be improved by a mile, and you are likely to get a nod. One reason for this issue is there is lack of common visibility. You need to see the same thing if you want to collaborate. Storage Team do not get always get access vSphere vSphere. Even if they do, vCenter UI is not designed for Storage team. It is designed for VMware Admin.

vRealize Operations and Log Insight can bridge that providing a set of read-only, purpose-built dashboards, that answer common questions such as:

  • When a VM Owner complains, can we clear if it’s a storage issue within 1 minute?
    • No ping pong between VM Owner, vSphere Admin, Storage Admin
  • Is the Storage serving all the VMs well?
    • If not, who are affected, when and how bad? Read or Write?
    • The answer has to be tier based, as Tier 1 VM expects lower latency than Tier 3
  • What’s the total demand hitting the array? Are they growing fast?
    • Who are the heavy hitters among the VMs?
  • When & where are we running out of capacity?
    • How much disk space can be reclaimed? From which VMs?
  • What have we got?
    • Are they consistently configured?

The questions above cover the main areas of SDDC Operations, such as performance, capacity, configuration and availability. They enable joint troubleshooting, capacity planning, performance monitoring. For better collaboration, add Blue Medora TVS, so you can analyze physical arrays and fabrics, and then correlate back with vSphere.

Overview

The first dashboard provides overall visibility to Storage team. It gives insight into the SDDC by showing relevant objects.

  • It quickly show the summary of key information.
  • It shows VM, datastore, datastore clusters, compute cluster, and datacenter. It shows their relationship, which you can interact and drill down.
  • It shows all the VMs, where they are located, how much space they are allocated, and how they are using it.

overview

Limitations & Customisation:

  • No RDM. Customers are moving away, and not many are using it to begin with.
  • Physical array Availability. This requires Storage MP or Blue Medora MP

Performance

The purpose of the vSphere platform is running VM. So long it is providing good service to the VM, we don’t have to explain the thing underneath. Whether 10,000 IOPS at VM level translates into 8,000 at hypervisor level due to caching at the host, is not as important as the VMs are being served well (as defined by SLA).

So we need to find out what’s the latency of every single VM in the vCenter. This is near impossible to establish in vCenter, as you have to go thru a lot of VMs. vR Ops helps using super metric.

Overall Performance

The set of dashboards answer questions such as:

  • What’s the overall performance, for each cluster and datastore? No point troubleshooting a VM or ESXi if the overall array is heavily hit.
  • What’s the total demand hitting the storage system? Who are the heavy hitters?
  • When a cluster is not performing, do we know when and which VMs were affected? Looking at cluster is useful as that’s where the demand comes from.

performance-overall

Total Latency is not Read + Write latency. In IP Storage, it is not Tx and Rx. It is both Tx as the ESXi host is sending the packets.

Datastore Performance

As this is for Storage Team, we can drill down to a specific datastore. It provides detail line charts of the datastore latency, throughput, outstanding IO and IOPS.

It also shows the VMs in the datastore, and if any of them is generating a lot of IOPS (villain VM).

performance-troubleshooting

Heavy Hitters

The performance problem could be caused by high overall loads. The dashboard shows you the total IOPS and total Throughput. If the number is high, you can drill down to see if there are Heavy Hitters.

What is Heavy Hitters? It has to be defined.

  1. What: IOPS or Throughput?
    • A VM with large block size can generate high throughput without doing excessive IOPS.
  2. How long: 5 minutes, 1 day, 1 week?
    • When we say heavy, how long before it becomes heavy?
    • The heavy hitters dashboard distinguishes between Bursty Hitter and Sustained Hitter.
  3. When: Today or yesterday? Any pattern?
    • The heavy hitters of yesterday may not be the heavy hitters of today.

It uses line chart so you can see pattern.

heavy-hitters

Limitation and Customisation:

  • Storage Team normally want sharper granularity than 5 minute. You can complement the above view VMkernel. I’ve shared posted here, so follow that link.
  • VM-level IOPS are different to Infra-level IOPS
    • Front-End IOPS are different to Back-End IOPS.
    • Distributed Storage has its own set back-end IOPS, such as synchronisation and replication.
  • We should know the cause of the poor storage performance.
    • The array?
      • Is it near its peak? Low cache, not enough spindle, etc.
      • Is it uneven (one SP doing the bulk of the work)? SP trespass?
      • Does it have hot spots?
    • The network or fabric?
      • Higher chance if you’re on IP Storage.
      • Happens if you don’t turn on Network QoS. There could be storage vMotion happening.
    • The ESX?
      • This could be bad configuration, like wrong multipath or insufficient HBA queue depth.
    • The VM?
      • Why does it generate high IOPS. Which process inside the Guest OS doing that?
      • A snapshot not removed?
      • A developer running IOmeter?

Capacity

The set of dashboards answer questions such as:

  • What’s the overall capacity? How is it used? Where are we on over-subscription?
  • Any datastore is running low on capacity? Are we using the datastores equally?
  • Are the VMs equally distributed among the datastores?
  • For each shared datastores, what are the capacity?

capacity-overall

The heat maps are useful in comparing across datastores:

  • Size by capacity
    • This shows if datastores are consistent in sizes. You should not have too many variance in size.
  • Color by capacity remaining.
    • Red = running out of capacity. Do not deploy more VM
    • Black = wastage. You’re not using it.
    • You should expect green.
  • Group by vCenter, then Datastore Cluster.
    • You know quickly where they are.

Combining the above 3 info into a single heat map tells you if you are using your storage properly.

You should use Datastore Cluster as part of your design. If you do, you also get visibility into their capacity, as shown below:

capacity-overall-datastore-cluster

You can drill down into each datastore and determine their capacity.

capacity-details-datastore

Limitation and Customisation:

  • Use datastore cluster. This should be part of your architecture anyway.
    • A limitation here is distributed storage, as it has no datastore clusters. You should exclude them and deal with them separately as the monitoring metrics are different.
  • If you have too many datastore clusters, then use Service Tier to group them

Capacity: Space Reclamation

There are 4 places where you can reclaim storage, from easiest to the hardest

  1. Orphaned VMs and orphaned VMDK. They are the easiest as they are not even owned.
  2. Snapshot.
  3. Powered Off VM
  4. Idle VM

capacity-reclamation

The dashboard does not list active VM because politically it’s hard to reclaim. Don’t bother trying 🙂

Storage as a Service

When a VM owner complain, can we rule out within 1 minute whether Storage is the issue?

Using the following dashboard, you select or browse for the VM in question. Its key storage properties and KPI will be automatically shown. We are using line chart as the problem might happen in the past and no longer present. You can also verify if it’s one off issue or regular issue.

The VM’s datastore will be shown automatically. The VM in the screenshot has its VMDK files in 3 different datastores. You can click on each, and the performance will be automatically shown. This lets you verify if the underlying datastore was able to cope or not.

performance-single-vm-troubleshooting

Limitation and Customisation:

  • The dashboard does not show Throughput. Throughput matters more on large block size. 4 – 32K block size should not be problem when IOPS is low.
  • The dashboard does not show Outstanding IO. This is useful to tell if underlying infra unable to process.
  • Add snapshot. Latency for snapshot will be higher as it has to go through multiple operations.

Hope you find the material useful. If you do, go back to the Main Page for the complete coverage of SDDC Operations. It gives you the big picture so you can see how everything fits together. If you already know how it all fits, you can go straight to download here.