The purpose of the eye-catching title is to drive a point that you need to look at the entire SDDC, and not just a component (e.g. Compute, Storage, Network, Security, Management, UPS). Once you look at the whole SDDC infrastructure in its entirety, you maybe surprised that you can shrink your footprint.
The purpose is not to say that you must achieve 1000 VMs per rack. It is also possible that you can’t even achieve 100 VMs per rack (for example, you are running all Monster VMs). I’m just using “visual” so it’s easier for you to see that there is a lot of inefficiency in typical data center. The number 1000 VM is an easy number to remember 🙂
If your entire data center shrinks into just 1 rack, what happens to the IT Organisation? You are right, it will have to shrink also.
- You may no longer need 3 separate teams (Architect, Implement, Operate).
- You may no longer need silos (Network, Server, Storage, Security).
- You may no longer need the layers (Admin, Manager, Director, Head)
With less people, there is less politics and the whole team becomes more agile.
The above is not just my personal opinion. Ivan Pepelnjak, a networking authority, has in fact shared back in October 2014 that “2000 VMs can easily fit onto 40 servers”. I recommend you review his calculation on this blog article. I agree with Ivan that “All you need are two top-of-rack switches” for your entire data center. Being a networking authority, he elaborates from networking angle. I’d like to complement it from a Server angle.
Let’s take a quick calculation to see how many VMs we can place in a standard 42 RU rack. I’d use Server VM, not Desktop VM, as they demand higher load.
I’d use a 2RU, 4 ESXi Host form factor, as this is a popular form factor. You can find example at SuperMicro site. Each ESXi has 2 Intel Xeon sockets and all flash local SSD running vSAN. With Intel Xeon E5-2699, each ESXi Host has 40 physical cores. Add 25% of Intel Hyper-Threading benefit, you can support ~30 VM with 3 vCPU on average as there are enough physical cores to schedule the VMs. Total 90 vCPU divided by 50 cores (40 + HT). This number is even better with Xeon Platinum.
The above take into account that a few cores are needed for:
- vSphere Replication
- NSX services from partners, which take the form of VM instead of kernel module.
30 VM for each ESXi. That’s 30:1 consolidation ratio, which is a reality today. You have 4 ESXi in a 2RU form factor. That means 30 x 4 = 120 VM fits into 2 RU space. Let’s assume you standardise on a 8-node cluster, and you do N+1 for HA. That means a cluster with HA will house 7 ESXi x 30 VM = 210 VMs. Each cluster only occupies 4 RU, and it comes with shared storage.
To hit ~1000 VMs, you just need 4+ clusters. In terms of rack space, that’s just 5 x 4 RU = 20 RU. Half a rack!
Let’s do ~1500 VM. This gives you 7 clusters. If you do 1000 VM that means you can have larger VM.
A standard rack has 42 RU. You still have 42 – 28 = 14 RU. That’s plenty of space for Networking, Internet connection, KVM, UPS, and Backup!
Networking will only take 2 x 2 RU. You can get 96 ports per 2 RU. Arista has models you can choose here. Yes, there is no need for spine-leaf architecture. That simplifies networking a lot.
KVM will only take 1 RU. With iLO, some customers do not use KVM as KVM encourages physical presence in data center.
If you still need a physical firewall, there is space for it.
If you prefer external storage, you can easily put 1400 VM into a 2RU all-flash storage. Tintri has an example here.
I’ve provided a sample rack design in this blog.
What do you think? How many racks do you still use to handle 1000 VM?
- [7 Nov 2015: Tom Carter spotted an area I overlooked. I forgot to take into account the power requirements! He was rightly disappointed, and this is certainly disappointing for me too, as I used to sell big boxes like Sun Fire 15K and HDS 9990! On big boxes like this, I had to ensure that customers data center has the correct cee form. Beyond just the Ampere, you need to know if they are single-phase or triple-phase. So Tom, thank you for the correction! Tom provided his calculation in Ivan’s blog, so please review it]
- [15 Nov 2015: Greg Ferro shared in his article that 1000 VM is certainly achievable. I agree with him that it’s a consideration. It’s not a goal nor a limit. It all depends on your application and situation]
- [27 Mar 2016: Intel Xeon E5-2699-V4 is delivering 22 cores per socket, up from 18 cores in v3]
- [16 July 2018: vSAN has wide adoption. Xeon Platinum has even more core, price of local SSD has gone down]