Building a Single-Node VSAN

I was wondering if it possible to speed up my Intel NUC based ESXi with Virtual SAN. The idea is that compared against vSphere Flash Read Cache, Virtual SAN can use the SSD not only as read cache but also as write buffer. This post explains how you can create a Virtual SAN Datastore on a single ESXi host from the command-line without a vCenter Server.

It goes without saying that this is neither the idea behind Virtual SAN nor officially supported by VMware. It also violates VMware's EULA if you are running Virtual SAN without a VSAN license. To assign a licence you need a vCenter Server and wrap the single ESXi into a Cluster.

My configuration for this test:

• 5th Gen Intel NUC NUC5i5MYHE
• 32GB Memory
• SSD: 250GB Samsung 850 Evo M.2
• HDD: 2.5" 1TB 5400rpm
• vSphere ESXi 6.0

The guide starts with a fresh ESXi installation in evaluation mode and I will not cover licensing. According to this post it also works with the free ESXi Hypervisor license, but to be compliant with VMware's EULA you have to add the host to a vCenter Server and assign a Virtual SAN license.

First step is to enable VSAN traffic for the VMkernel interface

~ # esxcli vsan network ipv4 add -i vmk0

Next step is to identify the devices to be used for VSAN. Both, SSD and HDD, must not contain any data or partitions. The easiest way to identify devices is with the vdq -q command.

In this example, the HDD already contains a VMFS partition:

~ # vdq -q
[
   {
      "Name"     : "t10.ATA_Samsung_SSD_850_EVO_M.2_250GB",
      "VSANUUID" : "",
      "State"    : "Eligible for use by VSAN",
"ChecksumSupport": "0",
      "Reason"   : "None",
      "IsSSD"    : "1",
"IsCapacityFlash": "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "t10.ATA_ST9100AS",
      "VSANUUID" : "",
      "State"    : "Ineligible for use by VSAN",
"ChecksumSupport": "0",
      "Reason"   : "Has partitions",
      "IsSSD"    : "0",
"IsCapacityFlash": "0",
      "IsPDL"    : "0",
   },

]

Use partedUtil to identify and delete all partions on ineligible drives. The partedUtil get command displays one partition (marked red) which can be deleted with the partedUtil delete command.

~ # partedUtil get /dev/disks/t10.ATA_ST9100AS
60801 255 63 976773168
1 2048 976768064 0 0

~ # partedUtil delete /dev/disks/t10.ATA_ST9100AS 1

Next step is to add physical disks for VSAN usage. VSAN requires one SSD and one, or multiple HDDs. Use the esxcli vsan storage add command with the options -d HDD1 -d HDD2 -s SDD

~ # esxcli vsan storage add -d t10.ATA_ST9100AS -s t10.ATA_Samsung_SSD_850_EVO_M.2_250GB

To enable VSAN on the host simply create a new Cluster.

~ # esxcli vsan cluster new

Now you should see the vsanDatastore, but creating Virtual Machines fails with the following error message:

Create virtual machine
Cannot complete file creation operation.

This is caused by a missing redundancy defined in default policies. To fix this, change the default policies to hostFailuresToTolerate = 0 and forceProvisioning = 1.

~ # esxcli vsan policy setdefault -c cluster -p "((\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i1))"
~ # esxcli vsan policy setdefault -c vdisk -p "((\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i1))"
~ # esxcli vsan policy setdefault -c vmnamespace -p "((\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i1))"
~ # esxcli vsan policy setdefault -c vmswap -p "((\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i1))"
~ # esxcli vsan policy setdefault -c vmem -p "((\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i1))"

You can verify the default policy with the following command:

~ # esxcli vsan policy getdefault

That's it. You can now use the Virtual SAN datastore on your single ESXi host.

~ # esxcli vsan cluster get
Cluster Information
  Enabled: true
  Current Local Time: 2015-11-09T10:26:45Z
  Local Node UUID: 563fc008-ab73-2e34-0075-005056bdaaf9
  Local Node State: MASTER
  Local Node Health State: HEALTHY
  Sub-Cluster Master UUID: 563fc008-ab73-2e34-0075-005056bdaaf9
  Sub-Cluster Backup UUID:
  Sub-Cluster UUID: 520afa76-164d-bab1-14ac-3cddab7a8570
  Sub-Cluster Membership Entry Revision: 0
  Sub-Cluster Member UUIDs: 563fc008-ab73-2e34-0075-005056bdaaf9
  Sub-Cluster Membership UUID: db744056-5f49-77a1-7f1d-005056bdaaf9

~ # esxcli vsan storage list
t10.ATA_ST9100AS
 Device: t10.ATA_ST9100AS
 Display Name: t10.ATA_ST9100AS
 Is SSD: false
 VSAN UUID: 5272476c-b445-4623-73f5-e2cd2a6265fd
 VSAN Disk Group UUID: 52b7be05-b875-a93e-24bc-62c2210948d7
 VSAN Disk Group Name: t10.ATA_Samsung_SSD_850_EVO_M.2_250GB
 Used by this host: true
 In CMMDS: false
 Checksum: 10071299334450500695
 Checksum OK: true
 Emulated DIX/DIF Enabled: false

t10.ATA_Samsung_SSD_850_EVO_M.2_250GB
 Device: t10.ATA_Samsung_SSD_850_EVO_M.2_250GB
 Display Name: t10.ATA_Samsung_SSD_850_EVO_M.2_250GB
 Is SSD: true
 VSAN UUID: 52b7be05-b875-a93e-24bc-62c2210948d7
 VSAN Disk Group UUID: 52b7be05-b875-a93e-24bc-62c2210948d7
 VSAN Disk Group Name: t10.ATA_Samsung_SSD_850_EVO_M.2_250GB
 Used by this host: true
 In CMMDS: false
 Checksum: 3233630865690359925
 Checksum OK: true
 Emulated DIX/DIF Enabled: false

I've done some basic performance tests with a datastore directely on the HDD, and with the VSAN datastore. Here are the results:

The test was the Exchange 2007 Workload from VMware I/O Analyzer (8k 55%Read 80%Random).

VMFS Datastore on local HDD
IOPS: 226
Read IOPS: 125
Write IOPS: 101
MBPS: 1.77
Read MBPS: 0.98
Write MBPS: 0.79

Single-Node VSAN Datastore
IOPS: 5294
Read IOPS: 2911
Write IOPS: 2382
MBPS: 41.36
Read MBPS: 22.75
Write MBPS: 18.61