1000 VM per rack is the new minimum

The purpose of the above article is to drive a point that you need to look at the entire SDDC, and not just a component (e.g. Compute, Storage, Network, Security, Management). Once you look at the whole SDDC infrastructure in its entirety, you maybe surprised that everything fits into just 1-2 rack!

The purpose is not to say that you must achieve 1000 VM per rack. It is also possible that you can’t even achieve 100 VM per rack (for example, you are running all Monster VMs). I’m just using “visual” so it’s easier for you to see that there is a lot of inefficiency in typical data center.

If your entire data center shrinks into just 1 rack, what happens to the IT Organisation? You are right, it will have to shrink also.

  • You may no longer need 3 separate team (Architect, Implement, Operate).
  • You may no longer need silos (Network, Server, Storage, Security).
  • You may no longer need the layers (Admin, Manager, Director, Head)

With less people, there is less politics and the whole team becomes more agile.

The above is not just my personal opinion. Ivan Pepelnjak, a networking authority, has in fact shared back in October 2014 that “2000 VMs can easily fit onto 40 servers”. I recommend you review his calculation on this blog article. I agree with Ivan that “All you need are two top-of-rack switches” for your entire data center. Being a networking authority, he elaborates from networking angle. I’d like to complement it from a Server angle.

Let’s take a quick calculation to see how many VMs we can place in a standard 42 RU rack. I’d use Server VM, not Desktop VM, as they demand higher load.

I’d use a 2RU, 4 ESXi Host form factor, as this is a popular form factor. You can find example at SuperMicro site. Each ESXi has 2 Intel Xeon sockets and all flash local SSD running distributed virtual storage. With Intel Xeon E5-2699 v3, each ESXi Host has 36 physical cores. Add 25% of Intel Hyper-Threading benefit, you can support ~30 VM with 2-3 vCPU each as there are enough physical cores to schedule the VMs.

The above take into account that a few cores are needed for

  • VMkernel
  • NSX
  • VSAN
  • vSphere Replication
  • NSX services from partners, which take the form of VM instead of kernel module.

30 VM for each ESXi. That’s 30:1 consolidation ratio, which is a reality today. You have 4 ESXi in a 2RU form factor. That means 30 x 4 = 120 VM fits into 2 RU space. Let’s assume you standardise on a 8-node cluster, and you do N+1 for HA. That means a cluster with HA will house 7 ESXi x 30 VM = 210 VMs. Each cluster only occupies 4 RU, and it comes with shared storage.

To hit ~1500 VMs, you just need 7 clusters. In terms of rack space, that’s just 7 x 4 RU = 28 RU.

Capture

A standard rack has 42 RU. You still have 42 – 28 = 14 RU. That’s plenty of space for Networking, Internet connection, KVM, UPS, and Backup!

Networking will only take 2 x 2 RU. You can get 96 ports per 2 RU. Arista has models you can choose here. Yes, there is no need for spine-leaf architecture. That simplifies networking a lot.

KVM will only take 1 RU. With iLO, some customers do not use KVM as KVM encourages physical presence in data center.

If you still need a physical firewall, there is space for it.

If you prefer external storage, you can easily put 1400 VM into a 2RU all-flash storage. Tintri has an example here.

I’ve provided a sample rack design in this blog.

What do you think? How many racks do you still use to handle 1000 VM?

Updates

  • [7 Nov 2015:  Tom Carter spotted an area I overlooked. I forgot to take into account the power requirements! He was rightly disappointed, and this is certainly disappointing for me too, as I used to sell big boxes like Sun Fire 15K and HDS 9990! On big boxes like this, I had to ensure that customers data center has the correct cee form. Beyond just the Ampere, you need to know if they are single-phase or triple-phase. So Tom, thank you for the correction! Tom provided his calculation in Ivan’s blog, so please review it]
  • [15 Nov 2015: Greg Ferro shared in his article that 1000 VM is certainly achievable. I agree with him that it’s a consideration. It’s not a goal nor a limit. It all depends on your application and situation]
  • [27 Mar 2016: Intel Xeon E5-2699-V4 is delivering 22 cores per socket, up from 18 cores in v3]