VMware vSAN on Consumer-Grade SSDs - Endurance analysis

When you are running an ESXi based homelab, you might have considered using vSAN as the storage technology of choice. Hyperconverged storages are a growing alternative to SAN-based systems in virtual environments, so using them at home will help to improve your skillset with that technology.

To get started with vSAN you need at least 3 ESXi Hosts, each equipped with 2 drives. Alternatively, you can build a 2-node vSAN Cluster using a Raspberry Pi as a witness node.

VMware maintains a special HCL that lists supported drives to be used with vSAN. In production setups, it is very important to use certified hardware. Using non-enterprise hardware might result in data loss and bad performance caused by the lack of power loss protection and small caches.

This article takes a look at consumer-grade SSDs and their durability to be used with vSAN. Please be aware that non-certified hardware should only be used in homelabs or for demo purposes. Do not place sensitive data on vSAN that is running on consumer hardware.

All-Flash or Hybrid vSAN?

I highly recommend to no longer use Hybrid vSAN in Homelabs. We are close to 2021 and magnetic disks are more and more vanishing. If you are planning to use vSAN you should go All-Flash.

Cache and Capacity Endurance (TBW)

There are two types of disks used in All-Flash vSAN - Capacity and Cache. In All-Flash, the cache layer is used for write-buffering only. Everything that is written to the vSAN Datastore is written to the cache and destaged later to the capacity tier. Currently, there is no way to run vSAN on a single disk, or omit the caching layer.

A couple of months ago, I've created a 2-node vSAN Cluster using two Intel NUCs and migrated the majority of my virtual machines from a shared storage to vSAN. This includes a vCenter Server, NSX-T components, Tanzu Kubernetes components, and a couple of management VMs. The see how vSAN affects my SSDs endurance (TBW) I've modified my NVMe TBW Monitoring Script to send data to Graphite.

With the collected metrics, I calculated how much GB is written per hour and day.
As shown in the graph, vSAN has written about 300 GB per day to the Caching Tier and 20GB per day to the Capacity tier. With this data, it is safe to say that you do not have to worry about the Capacity Tier's endurance. Do get an idea of what 300GB/day means for the SSDs endurance, I did some calculations:

  • 3.5 MB per second
  • 213 MB per minute
  • 12.5 GB per hour
  • 300 GB per day
  • 8.8 TB per month
  • 107 TB per year
  • 535 TB in 5 years

When buying SSDs, the vendor gives a guarantee in years and "TBW" (Terabytes Written). With most vendors, the guarantee expires when either 5 years have passed, or more data than specified in TBW has been written to the drive.

With my vSAN setup, I need an SSD with at least 535 TBW, not calculation any growth.

Choosing SSD for vSAN

When choosing SSDs for vSAN you typically would buy a small SSD for caching, and a large SSD for the capacity tier. When using consumer hardware, it might be a good idea to also use a large SSD for caching, as the endurance grows with the size of the device.

With the data gathered above, I would need an SSD that has a TBW endurance of at least 535 TB. The following table shows capacity and TBW values of commonly used consumer grade SSDs:

VendorProduct NameCapacityTBWWarranty
Samsung980 PRO NVMe M.2 SSD2501505 years
Samsung980 PRO NVMe M.2 SSD5003005 years
Samsung980 PRO NVMe M.2 SSD10006005 years
Samsung950 PRO NVMe M.2 SSD2562005 years
Samsung950 PRO NVMe M.2 SSD5124005 years
SamsungSSD 870 QVO SATA III 2.5 Inch10003603 years
SamsungSSD 870 QVO SATA III 2.5 Inch20007203 years
SamsungSSD 870 QVO SATA III 2.5 Inch400014403 years
SamsungSSD 870 QVO SATA III 2.5 Inch800028803 years
Samsung970 EVO Plus NVMe M.2 SSD2501505 years
Samsung970 EVO Plus NVMe M.2 SSD5003005 years
Samsung970 EVO Plus NVMe M.2 SSD10006005 years
Samsung970 EVO Plus NVMe M.2 SSD200012005 years
Samsung860 QVO SATA III 2.5 Inch SSD10003603 years
Samsung860 QVO SATA III 2.5 Inch SSD20007203 years
Samsung860 QVO SATA III 2.5 Inch SSD400014403 years
Samsung970 EVO NVMe M.2 SSD2501505 years
Samsung970 EVO NVMe M.2 SSD5003005 years
Samsung970 EVO NVMe M.2 SSD10006005 years
Samsung970 EVO NVMe M.2 SSD200012005 years
Samsung970 PRO NVMe M.2 SSD5126005 years
Samsung970 PRO NVMe M.2 SSD100012005 years
Samsung860 EVO SATA III 2.5 Inch SSD2501505 years
Samsung860 EVO SATA III 2.5 Inch SSD5003005 years
Samsung860 EVO SATA III 2.5 Inch SSD10006005 years
Samsung860 EVO SATA III 2.5 Inch SSD200012005 years
Samsung860 EVO SATA III 2.5 Inch SSD400024005 years
SamsungSSD 860 PRO SATA III 2.5 Inch2563005 years
SamsungSSD 860 PRO SATA III 2.5 Inch5126005 years
SamsungSSD 860 PRO SATA III 2.5 Inch100012005 years
SamsungSSD 860 PRO SATA III 2.5 Inch200024005 years
SamsungSSD 860 PRO SATA III 2.5 Inch400048005 years
Samsung860 EVO SATA III M.2 SSD2501505 years
Samsung860 EVO SATA III M.2 SSD5003005 years
Samsung860 EVO SATA III M.2 SSD10006005 years
Samsung860 EVO SATA III M.2 SSD200012005 years
IntelIntel Optane Memory 16GB16182.55 years
IntelIntel Optane Memory M10 16GB163655 years
IntelIntel Optane Memory 32GB32182.55 years
IntelIntel Optane Memory M10 32GB323655 years
IntelIntel Optane SSD 800P 58GB583655 years
IntelIntel Optane SSD 800P 58GB1183655 years
IntelIntel® 545s-SSDs2561445 years
IntelIntel® 545s-SSDs5122885 years
IntelIntel® 545s-SSDs2561445 years
IntelIntel® 660p-SSDs5121005 years
IntelIntel® 660p-SSDs10242005 years
IntelIntel® 660p-SSDs20484005 years
IntelIntel® 760p-SSDs128725 years
IntelIntel® 760p-SSDs2561445 years
IntelIntel® 760p-SSDs5122885 years
IntelIntel® 760p-SSDs10245765 years
IntelIntel® 760p-SSDs20485765 years
TranscendPCIe SSD 220S2565505 years
TranscendPCIe SSD 220S51211005 years
TranscendPCIe SSD 220S100022005 years
TranscendPCIe SSD 220S200044005 years
SeagateIronWolf 5102404355 years
SeagateIronWolf 5104808755 years
SeagateIronWolf 51096017505 years
SeagateIronWolf 510192035005 years

 

7 thoughts on “VMware vSAN on Consumer-Grade SSDs - Endurance analysis”

  1. Hi Florian,
    Can you please show your configuration from your homelab, Screenshots from the vcenter to see, which hardware is in your esx...
    Thx
    frAnk

    1. The other thing to remember is that QVO drives will really slow down once their cache is full. The EVO drives are able to maintain speed.

      So...I used 2 TB QVO drives for capacity and a 1 TB EVO drive for the cache. Should provide a good balance of speed, capacity and price.

    2. only problem is that now that 7.0.3 is out, it does not allow you to use UM/Lifecycle Manager to install updates at all if the NVMe drives are not certified. I have been using 960 Pro NVMe drives for a couple of years but once I installed 7.0.3 it is failing big time. First it did not even identify them, so I told it they were SM963 which is what it has been identifying them as from the beginning. But once it installed 7.0.3, it now sees them properly as 960 Pro drives and won't let me update it again as these drives are not on the compatibility list and have never been on the compatibility list.

Leave a Reply to frAnk Cancel reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.