VMware vSAN on Consumer-Grade SSDs - Endurance analysis

When you are running an ESXi based homelab, you might have considered using vSAN as the storage technology of choice. Hyperconverged storages are a growing alternative to SAN-based systems in virtual environments, so using them at home will help to improve your skillset with that technology.

To get started with vSAN you need at least 3 ESXi Hosts, each equipped with 2 drives. Alternatively, you can build a 2-node vSAN Cluster using a Raspberry Pi as a witness node.

VMware maintains a special HCL that lists supported drives to be used with vSAN. In production setups, it is very important to use certified hardware. Using non-enterprise hardware might result in data loss and bad performance caused by the lack of power loss protection and small caches.

This article takes a look at consumer-grade SSDs and their durability to be used with vSAN. Please be aware that non-certified hardware should only be used in homelabs or for demo purposes. Do not place sensitive data on vSAN that is running on consumer hardware.

All-Flash or Hybrid vSAN?

I highly recommend to no longer use Hybrid vSAN in Homelabs. We are close to 2021 and magnetic disks are more and more vanishing. If you are planning to use vSAN you should go All-Flash.

Cache and Capacity Endurance (TBW)

There are two types of disks used in All-Flash vSAN - Capacity and Cache. In All-Flash, the cache layer is used for write-buffering only. Everything that is written to the vSAN Datastore is written to the cache and destaged later to the capacity tier. Currently, there is no way to run vSAN on a single disk, or omit the caching layer.

A couple of months ago, I've created a 2-node vSAN Cluster using two Intel NUCs and migrated the majority of my virtual machines from a shared storage to vSAN. This includes a vCenter Server, NSX-T components, Tanzu Kubernetes components, and a couple of management VMs. The see how vSAN affects my SSDs endurance (TBW) I've modified my NVMe TBW Monitoring Script to send data to Graphite.

With the collected metrics, I calculated how much GB is written per hour and day.
As shown in the graph, vSAN has written about 300 GB per day to the Caching Tier and 20GB per day to the Capacity tier. With this data, it is safe to say that you do not have to worry about the Capacity Tier's endurance. Do get an idea of what 300GB/day means for the SSDs endurance, I did some calculations:

• 3.5 MB per second
• 213 MB per minute
• 12.5 GB per hour
• 300 GB per day
• 8.8 TB per month
• 107 TB per year
• 535 TB in 5 years

When buying SSDs, the vendor gives a guarantee in years and "TBW" (Terabytes Written). With most vendors, the guarantee expires when either 5 years have passed, or more data than specified in TBW has been written to the drive.

With my vSAN setup, I need an SSD with at least 535 TBW, not calculation any growth.

Choosing SSD for vSAN

When choosing SSDs for vSAN you typically would buy a small SSD for caching, and a large SSD for the capacity tier. When using consumer hardware, it might be a good idea to also use a large SSD for caching, as the endurance grows with the size of the device.

With the data gathered above, I would need an SSD that has a TBW endurance of at least 535 TB. The following table shows capacity and TBW values of commonly used consumer grade SSDs: