Monthly Archives: October 2015

1000 VM per rack is the new minimum

The purpose of the eye-catching title is to drive a point that you need to look at the entire SDDC, and not just a component (e.g. Compute, Storage, Network, Security, Management, UPS). Once you look at the whole SDDC infrastructure in its entirety, you maybe surprised that you can shrink your footprint.

The purpose is not to say that you must¬†achieve 1000 VMs per rack. It is also possible that you can’t even achieve 100 VMs per rack (for example, you are running all Monster VMs). I’m just using “visual” so it’s easier for you to see that there is a lot of inefficiency in typical data center. The number 1000 VM is an easy number to remember ūüôā

If your entire data center shrinks into just 1 rack, what happens to the IT Organisation? You are right, it will have to shrink also.

  • You may no longer need 3 separate teams (Architect, Implement, Operate).
  • You may no longer need silos (Network, Server, Storage, Security).
  • You may no longer need the layers (Admin, Manager, Director, Head)

With less people, there is less politics and the whole team becomes more agile.

The above is not just my personal opinion. Ivan Pepelnjak, a networking authority, has in fact shared back in October 2014 that ‚Äú2000 VMs can easily fit onto 40 servers‚ÄĚ. I recommend you review his¬†calculation on this blog article. I agree with Ivan that “All you need are two top-of-rack switches” for your entire data center. Being a networking authority, he elaborates from networking angle. I’d like to complement it from a Server angle.

Let’s take a quick calculation to see how many VMs we can place in a standard 42 RU rack. I’d use Server VM, not Desktop VM, as they demand higher load.

I’d use a 2RU, 4 ESXi Host form factor, as this is a popular form factor. You can find example at SuperMicro site. Each ESXi has 2 Intel Xeon sockets and all flash local SSD running vSAN. With Intel Xeon E5-2699, each ESXi Host has 40 physical cores.¬†Add 25% of Intel Hyper-Threading benefit, you can support ~30 VM with 3 vCPU on average¬†as there are enough physical cores to schedule the VMs. Total 90 vCPU divided by 50 cores (40 + HT). This number is even better with Xeon Platinum.

The above take into account that a few cores are needed for:

  • VMkernel
  • NSX
  • vSAN
  • vSphere Replication
  • NSX services from partners, which take the form of VM instead of kernel module.

30 VM for each ESXi. That’s 30:1 consolidation ratio, which is a reality today. You have 4 ESXi in a 2RU form factor. That means 30 x 4 = 120 VM fits into 2 RU space. Let’s assume you standardise on a 8-node cluster, and you do N+1 for HA. That means a cluster with HA will house 7 ESXi x 30 VM = 210 VMs. Each cluster only occupies 4 RU, and it comes with shared storage.

To hit ~1000 VMs, you just need 4+ clusters. In terms of rack space, that’s just 5 x 4 RU = 20 RU. Half a rack!

Let’s do ~1500 VM. This gives you 7 clusters. If you do 1000 VM that means you can have larger VM.


A standard rack has 42 RU. You still have 42 – 28 = 14 RU. That’s plenty of space for Networking, Internet connection, KVM, UPS, and Backup!

Networking will only take 2 x 2 RU. You can get 96 ports per 2 RU. Arista has models you can choose here. Yes, there is no need for spine-leaf architecture. That simplifies networking a lot.

KVM will only take 1 RU. With iLO, some customers do not use KVM as KVM encourages physical presence in data center.

If you still need a physical firewall, there is space for it.

If you prefer external storage, you can easily put 1400 VM into a 2RU all-flash storage. Tintri has an example here.

I’ve provided a sample rack design¬†in this blog.

What do you think? How many racks do you still use to handle 1000 VM?


  • [7 Nov 2015:¬† Tom Carter spotted an area I overlooked. I forgot to take into account the power requirements! He was rightly disappointed, and this is certainly disappointing for me too, as I used to sell big boxes like Sun Fire 15K and HDS 9990! On big boxes like this, I had to ensure that customers data center has the correct cee form. Beyond just the Ampere, you need to know if they are single-phase or triple-phase. So Tom, thank you for the correction! Tom provided his calculation in Ivan’s blog, so please review it]
  • [15 Nov 2015:¬†Greg Ferro shared in his article that 1000 VM is certainly achievable. I agree with him that it’s a consideration. It’s not a goal nor a limit. It all depends on your application and situation]
  • [27 Mar 2016: Intel Xeon E5-2699-V4 is delivering 22 cores per socket, up from 18 cores in v3]
  • [16 July¬†2018: vSAN has wide adoption. Xeon Platinum has even more core, price of local SSD has gone down]

Indonesian Cloud sharing at vForum

Glad to know that Indonesian Cloud is sharing how they use VMware technology in vForum Indonesia. Every Enterprise IT wants to be a Private Cloud provider, and becomes a business service provider instead of technology system builder. To me, Indonesian Cloud is a good success story of how you take advantage of VMware technology and partnership.

Neil Creswell, their CEO, is doing the keynote. Yes, that link is his Twitter handle. As you can see, he is engaging customers on social media. To me, if you are an IT leader you need to be engaging in social media. Hiding behind the comfort of office wall is not what your customers want. This is part of the Digital Business imperative. This is even more so if cloud is a key component of your business. Social media provides transparency that customers and partners value.

Speaking of transparency, price certainly comes to mind. Your customers want transparency. Below is what Indonesian Cloud shares openly.


Can you spot something assuring from the above price list? Something that is the hallmark of transparency.

Yes. No Bill Shock.

I heard it from customers again and again. They got a bill shock from their cloud provider. How often do you get a bill shock? Business is doing poor, yet your cloud cost increases? A common story is you get hit by a bug or problem, which generates excessive logs or consumes a lot of network. You get a high bill for that!

In the cloud, cost has to be simple. Ideally, it should be like your utility bill. Today’s cloud pricing is complex, with a lot of add-on. Indonesian Cloud’s pricing is simple and predictable, something we can learn from.

One factor that drives simplicity is the technology stack. Indonesian Cloud runs almost all the VMware stack. On the hardware powering the ESXi, they run a hyper-converged from SuperMicro, as shared by VMware top blogger Duncan Epping in his blog. That 2RU, 4 Node form factor provides high density setup. On top of that, they use VMware VSAN.


A customer once told me, “To build is one thing, to operate is another”. Operations, what happens after you live, is certainly essential. Indonesian Cloud has a Performance SLA concept. They know when any customer’s VM is experiencing degradation, often before the customer itself notice that. They are using the vRealize Operations contention counter to track it. They are also using vRealize Log Insight to analyse the log files. The team uses Log Insight to analyse not just vSphere, not just VMware, but also non VMware products.

I had the joy of working with their team in creating some creative dashboards to help them monitor their environment. They’ve allowed me to share one of the dashboard here.


I worked closely with Indonesian Cloud since March 2015. I still keep the email from Neil, which started the journey of partnership. I knew Neil for many years, so it was an easy decision for me to partner with his awesome team. He was right, his team was both dedicated and sharp. We came up with several innovative dashboards in both vRealize Operations and Log Insight. A lot of days¬†were spent discussing how to best monitor a VMware-powered SDDC, and I’m glad that the result speak for itself.

There is a lot that other VMware customers and Cloud Providers can learn from Indonesian Cloud. Being a Cloud Provider, be it private or public, is complex but doable. So it’s good to know that Neil is sharing it with fellow customers/partners. If I were a CIO, I’d ask for a site visit to get the insight. The NOC is a good place as you can see it live.

How to update VMware vSphere Update Manager 6

I could not find an article on how to update VMware vSphere Update Manager. There are many articles on how to do a fresh install. I found these articles from VMwareAndMe and vladan to be clear and informative. There is also a good set of tips from one of my favourite source, Notes from Michael White, which you can find here.

The update does not show the version number as 6.0 Update 1. Instead, it’s showing as¬†6.0.027278. The Update Manager for vSphere 6.0 build no is

To start, download the Windows ISO. It is not a separate install. It is part of the vCenter. Once downloaded, mount the ISO image from the Windows VM where your Update Manager is installed.

Stop the Update Manager service from Windows services. I’m using Windows 2008 here.


Do not tick the box for the Embedded Database Option, as you already have a database. Click Install button.


Don’t worry that it does not the Upgrade option.¬†It will detect that you have existing instance, as you can see below.


It will inform you that it will update the current version to build 27278. It does not say Update Manager 6.0 Update 1.


[8 April 2016 update: The same case with vSphere 6 Update 2. It will only say build 29963]

You get the usual prompt that it was installed successfully. The main installation window was not closed automatically, as you may want to install other thing.

It does not start the service in my case, so I just started it manually. No need to reboot Windows.¬†Close your Update Manager VM server. You’re done with this VM ūüôā

Go to your Admin Clien. Update the vSphere C# client plug-in. It should detect that you have the latest version. This time around, the installer shows you Update 1.


And that’s it! Login to your Web Client, and you should see all your Update Managers. I have 2 Update Managers, and both are listed. Interestingly, one was shown as the IP address.


To use the new UI, head over to this blog.

BTW, if you are using HP Proliant Server, and wants to integrate HP add-ons, head to a good write up here.

Happy updating!