Working with FreeNAS 9.3 Block VAAI Primitives

FreeNAS 9.3 now fully supports all 7 VAAI (vStorage APIs for Array Integration) block primitives:

  • Atomic Test & Set (ATS / Hardware Accelerated Locking)
  • XCOPY (Extended Copy / Clone)
  • Write Same (Zero)
  • Dead Space Reclamation (Delete / UNMAP)
  • Thin Provisioning LUN Reporting
  • Thin Provisioning Space Threshold Warning
  • Thin Provisioning Stun

Atomic Test & Set (ATS / Hardware Accelerated Locking)

ATS is a storage based locking mechanism to replace SCSI reservations on VMFS volumes when doing metadata updates. A SCSI reservation locks a datastore and prevents all other hosts from doing metadata updates on that datastore.

ATS is enabled by default on all FreeNAS iSCSI LUNS

Verify ATS support with esxcli storage core device vaai status get

[root@esx1:~] esxcli storage core device vaai status get -d naa.6589cfc0000006948963c2aa5057083f
naa.6589cfc0000006948963c2aa5057083f
   VAAI Plugin Name:
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: supported

Verify ATS operations with esxtop. Press u f f g i o in esxtop. This will enable Device mode and all required fields.
vaai-ats-in-esxtop

  • ATS - Total number of ATS commands
  • ATSF - Total number of failed ATS commands.

 

XCOPY (Extended Copy / Clone)

Without VAAI clone and Storage vMotion operations are performed by the VMkernel. The XCOPY primitive allows to offload these requests to the storage array. This does not waste ESXi Host resources such as CPU cycles, DMA buffers and SCSI commands in the HBA queue.

XCOPY is enabled by default on all FreeNAS iSCSI LUNS

Verify XCOPY support with esxcli storage core device vaai status get

[root@esx1:~] esxcli storage core device vaai status get -d naa.6589cfc0000006948963c2aa5057083f
naa.6589cfc0000006948963c2aa5057083f
   VAAI Plugin Name:
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: supported

Verify XCOPY operations with esxtop. Press u f f g i o in esxtop. This will enable Device mode and all required fields. A Virtual Machine was migrated with Storage DRS while the screenshot was taken.
vaai-xcopy-in-esxtop

  • Clone_RD - Total number of reads offloaded
  • Clone_WR - Total number of writes offloaded
  • Clone_F - Total number of failed CXOPY commands
  • MBC_RD/s - Current megabytes per second read
  • MBC_WR/s - Current magabytes per second written

 

Write Same (Zero)

When an eager-zeroed virtual disk is created, the entire space gets zeroed. The ESXi host has to write gigabytes of zeroes to the storage array which takes some time. By using the Write Same primitive, the ESXi host simply tells the storage array how much zeroes it should write. This would significantly shorten the time needed.

Write Same is enabled by default on all FreeNAS iSCSI LUNS

Verify Write Same support with esxcli storage core device vaai status get

[root@esx1:~] esxcli storage core device vaai status get -d naa.6589cfc0000006948963c2aa5057083f
naa.6589cfc0000006948963c2aa5057083f
   VAAI Plugin Name:
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: supported

Verify Write Same operations with esxtop. Press u f f g i o in esxtop. This will enable Device mode and all required fields. An eager-zeroed virtual disk was created while the screenshot was taken.
vaai-zero-in-esxtop

  • Zero - Total number of ZERO commands offloaded
  • Zero_F - Total number of failed ZERO commands
  • MBZERO/s - Current megabytes per second zeroed

 

Dead Space Reclamation (UNMAP)

When a Virtual Machine on a thin-provisioned LUN is deleted, the storage array does not know that the blocks are unused, so it reports the space still as occupied. The UNMAP command enables an ESXi host to inform the storage array that space from a thin provisioned LUN can be reclaimed. This enables an array to reclaim space of a thin-provisioned datastore.

Dead Space Reclamation does only work if your iSCSI extent is a ZVOL created as sparse volume:
freenas-sparse-zvol

Verify Dead Space Reclamation support with esxcli storage core device vaai status get

[root@esx1:/vmfs/volumes/54b2d835-32f3bccf-e9d2-001b2193b3b0] esxcli storage core device vaai status get -d naa.6589cfc0000006948963c2aa5057083f
naa.6589cfc0000006948963c2aa5057083f
   VAAI Plugin Name:
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: supported

Dead Space is not automatically reclaimed.  The unmap process must be triggered manually:

[root@esx1:~] vmkfstools -y /vmfs/volumes/esxi-sparse/
Try to unmap 152372 blocks in units of 200 blocks from volume /vmfs/volumes/esxi-sparse/.

Verify Dead Space Reclamation with esxtop. Press u f f g i o in esxtop.
vaai-delete-in-esxtop

  • Delete - Total number of UNMAP commands
  • Delete_F - Total number of failed UNMAP commands
  • MBDEL/s - Current megabytes per second reclaimed

 

VAAI Thin Provisioning Primitives

Thin Provisioning LUN Reporting
A simple VAAI primitive that allows an ESXi Host to determine if a LUN is thin-provisioned. Determine LUN status with esxcli storage core device list

[root@esx1:~] esxcli storage core device list -d naa.6589cfc0000006948963c2aa5057083f |grep Thin
 Thin Provisioning Status: yes

Thin Provisioning Space Threshold Warning
With the this VAAI primitive, a warning is displayed in the Event Log if a thin-provisioned datastore reaches a specific threshold. The threshold can be set in the extent configuration:
freenas-threshold-sparse-zvol
Thin Provisioning Stun
Without this VAAI primitive, all virtual machines would crashes if a datastore is full. With VAAI Stun, only those virtual machines requiring extra blocks of storage space are paused. After additional space is allocated to the thin-provisioned datastore, the paused virtual machines can be resumed.

  1. This phrase is not true: "Dead Space Reclamation does only work if your iSCSI extent is a ZVOL created as sparse volume".

    UNMAP works independently of this checkbox status. But setting that checkbox is much less useful without UNMAP support, since sooner or later LUN will occupy space equal to its size, and then this checkbox loose any meaning.

    This checkbox set allows storage over-provisioning, when administrator may create LUN bigger then whole pool capacity. And thanks to UNMAP and space thresholds that can be really useful.

    • Thanks for clarification. I can run UMAP but what does actually happen on a ZVOL without the sparse volume checkbox? Do I gain of an advantage? It doesn't shrink and I do not get space back in the pool.
      It's true that it works, but what for?

      • Alexander Motin

        UNMAP'ed space is returned back to ZFS pool. You may see it as FREE in `zpool list` output. While ZFS won't let you use that space for other things due to reservation set on ZVOL, more free space on the pool helps to reduce fragmentation. Also, if you create a snapshot of ZVOL, it won't reference UNMAP'ed data, consuming less space when ZVOL will later get overwritten.

      • Alexander Motin

        It is called "Resource Provisioning". It is like UNMAP for SSDs -- does not give you any space back, but makes device live longer and happier.

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>