Manage Virtual SAN 5.5 with RVC - Complete Guide

This is a comprehensive guide to manage your VMware Virtual SAN with the Ruby vSphere Console. RVC is an interactive command line tool to control your platform. If you are new to RVC, make sure to read the Getting Started with Ruby vSphere Console Guide. All commands are from the latest vSphere 5.5 Update2 release.

  1. Preparation
  2. Basic Configuration
  3. Cluster Administration
  4. Object Management
  5. Troubleshooting
  6. VSAN Observer

 

Preparation

To make commands look clearer, I created marks for a Cluster, a Virtual Machine and an ESXi Hosts. This allows me to use ~cluster, ~vm and ~esx in my examples:

/localhost/DC> mark cluster ~/computers/VSAN-Cluster/
/localhost/DC> mark vm ~/vms/vma.virten.local
/localhost/DC> mark esx ~/computers/VSAN-Cluster/hosts/esx1.virten.local/

Basic Configuration

vsan.enable_vsan_on_cluster
Enables VSAN on a cluster. When VSAN is already enabled you can also use this command to enable/disable auto claim. When not specified, auto claim is enabled by default.

usage: enable_vsan_on_cluster [opts] cluster
Enable VSAN on a cluster
  cluster: Path to a ClusterComputeResource
  --disable-storage-auto-claim, -d: Disable auto disk-claim
                        --help, -h: Show this message

Example 1 - Enable VSAN on a Cluster:

/localhost/DC> vsan.enable_vsan_on_cluster ~cluster
ReconfigureComputeResource VSAN-Cluster: success
 esx1.virten.local: success
 esx2.virten.local: success
 esx3.virten.local: success

Example 2 - Enable VSAN on a Cluster with auto claim disabled, or disables auto claim on a VSAN enabled Cluster:

/localhost/DC> vsan.enable_vsan_on_cluster --disable-storage-auto-claim ~cluster
ReconfigureComputeResource VSAN-Cluster: success
 esx1.virten.local: success
 esx2.virten.local: success
 esx3.virten.local: success

vsan.disable_vsan_on_cluster
Disables VSAN on a cluster. VMware HA has to be disabled, to disable Virtual SAN. This command will make the vsanDatastore including all virtual machines unavailable. It does not delete any data.

/localhost/DC> vsan.disable_vsan_on_cluster --help
usage: disable_vsan_on_cluster cluster
Disable VSAN on a cluster
 cluster: Path to a ClusterComputeResource
 --help, -h: Show this message

Example 1 - Disable VSAN on a HA enabled Cluster:

/localhost/DC> cluster.configure_ha --disabled ~cluster
ReconfigureComputeResource VSAN-Cluster: success
 esx1.virten.local: success
 esx2.virten.local: success
 esx3.virten.local: success
/localhost/DC> vsan.disable_vsan_on_cluster ~cluster
ReconfigureComputeResource VSAN-Cluster: success
 esx1.virten.local: success
 esx2.virten.local: success
 esx3.virten.local: success

vsan.cluster_change_autoclaim
This command enables or disables auto-claim. With auto-claim enabled all unused disks are automatically consumed by Virtual SAN.

usage: cluster_change_autoclaim [opts] cluster
Enable VSAN on a cluster
  cluster: Path to a ClusterComputeResource
   --enable, -e: Enable auto-claim
  --disable, -d: Disable auto-claim
     --help, -h: Show this message

Example 1 - Enable auto-claim:

/localhost/DC> vsan.cluster_change_autoclaim --enable ~cluster
ReconfigureComputeResource VSAN-Cluster: success
 esx2.virten.local: success
 esx3.virten.local: success
 esx1.virten.local: success

Example 2 - Disable auto-claim:

/localhost/DC> vsan.cluster_change_autoclaim --disable ~cluster
ReconfigureComputeResource VSAN-Cluster: success
 esx2.virten.local: success
 esx3.virten.local: success
 esx1.virten.local: success

 

vsan.apply_license_to_cluster
Applies a Virtual SAN license key to a Cluster. This command also triggers a Null-Reconfigure on all hosts to ensure that all disks are claimed when auto-claim is enabled.

usage: apply_license_to_cluster [opts] cluster
Apply license to VSAN
  cluster: Path to a ClusterComputeResource
   --license-key, -k <s>: License key to be applied to the cluster
  --null-reconfigure, -r: (default: true)
              --help, -h: Show this message

Example 1 - Apply a VSAN license and for auto-claim:

/localhost/DC> vsan.apply_license_to_cluster -k 00000-00000-00000-00000-00000 ~cluster
VSAN-Cluster: Applying VSAN License on the cluster...
VSAN-Cluster: Null-Reconfigure to force auto-claim...
ReconfigureComputeResource VSAN-Cluster: success
 esx1.virten.local: success
 esx2.virten.local: success
 esx3.virten.local: success

Example 2 - Replace a VSAN license key:

/localhost/DC> vsan.apply_license_to_cluster -r -k 00000-00000-00000-00000-00000 ~cluster 
VSAN-Cluster: Applying VSAN License on the cluster...

vsan.host_consume_disks
Consumes all eligible disks on a host. When you have multiple SSDs you can specify the appropriate SSD by its model.

usage: host_consume_disks [opts] host_or_cluster...
Consumes all eligible disks on a host
  host_or_cluster: Path to a ComputeResource or HostSystem
  --filter-ssd-by-model, -f <s>: Regex to apply as ssd model filter
                     --help, -h: Show this message

Example 1 - Identify SSD Model description and create a disk group. This is required when you have more than one SSD:

/localhost/DC> esxcli.execute ~esx storage core device list
[...]
IsSSD: true
Model: "SanDisk SDSSDP06"
[...]
/localhost/DC> vsan.host_consume_disks ~esx --filter-ssd-by-model "SanDisk SDSSDP06"
 AddDisks esx1.virten.local: success

Example 2 - Add all eligible disks to a new or existing disk group:

/localhost/DC> vsan.host_consume_disks ~esx
 AddDisks esx1.virten.local: success

vsan.cluster_set_default_policy
Set default policy on a cluster. That policy is used by default when an object has no VM Storage Policy defined. A policy defines how objects are stored in the Virtual SAN. Available values are:

  • hostFailuresToTolerate (Number of failures to tolerate)
  • forceProvisioning (If VSAN can't fulfill the policy requirements for an object, it will still deploy it)
  • stripeWidth (Number of disk stripes per object)
  • cacheReservation (Flash read cache reservation)
  • proportionalCapacity (Object space reservation)

Default Policy: (("hostFailuresToTolerate" i1))

usage: cluster_set_default_policy cluster policy
Set default policy on a cluster
  cluster: Path to a ClusterComputeResource
  policy:
  --help, -h: Show this message

Example 1 - Change the default policy to tolerate two host failures:

/localhost/DC> vsan.cluster_set_default_policy ~cluster '(("hostFailuresToTolerate" i2))'

Example 2 - Determine the current default policy from an existing host and add a rule:

/localhost/DC> esxcli.execute ~esx vsan policy getdefault
+--------------+----------------------------------------------------------+
| Policy Class | Policy Value                                             |
+--------------+----------------------------------------------------------+
| cluster      | (("hostFailuresToTolerate" i1))                          |
| vdisk        | (("hostFailuresToTolerate" i1))                          |
| vmnamespace  | (("hostFailuresToTolerate" i1))                          |
| vmswap       | (("hostFailuresToTolerate" i1) ("forceProvisioning" i1)) |
+--------------+----------------------------------------------------------+
/localhost/DC> vsan.cluster_set_default_policy ~cluster '(("hostFailuresToTolerate" i1) ("stripeWidth" i2))'

vsan.host_wipe_vsan_disks
Deletes the content of all VSAN enabled disks on a host. When the command is uses without the force option it only tells what it would wipe.

usage: host_wipe_vsan_disks [opts] host...
Wipes content of all VSAN disks on a host
  host: Path to a HostSystem
  --force, -f: Apply force
   --help, -h: Show this message

Example 1 - Run the command in dry mode:

/localhost/DC> vsan.host_wipe_vsan_disks ~esx
Would wipe disk WDC_WD1500HLFS (ATA WDC WD1500HLFS-0, ssd = false)
Would wipe disk WDC_WD1500HLFS (ATA WDC WD1500HLFS-0, ssd = false)
Would wipe disk WDC_WD1500HLFS (ATA WDC WD1500HLFS-0, ssd = false)
Would wipe disk SanDisk_SDSSDP064G (ATA SanDisk SDSSDP06, ssd = true)

NO ACTION WAS TAKEN. Use --force to actually wipe.
CAUTION: Wiping disks means all user data will be destroyed!

Example 2 - Wipe the content of all VSAN disks:

/localhost/DC> vsan.host_wipe_vsan_disks ~esx --force
 RemoveDiskMapping esx1.virten.local: success

Cluster Administration

vsan.host_info
Prints VSAN related information about an ESXi host. This command retains information about:

  • Cluster role (master, backup or agent)
  • Cluster UUID
  • Node UUID
  • Member UUIDs
  • Auto claim (yes or no)
  • Disk Mapptings: Disks that are claimed by VSAN
  • NetworkInfo: VSAN traffic activated vmk adapters
usage: host_info host
Print VSAN info about a host
  host: Path to a HostSystem
  --help, -h: Show this message

Example 1 - Print VSAN host Information:

/localhost/DC> vsan.host_info ~esx
VSAN enabled: yes
Cluster info:
  Cluster role: agent
  Cluster UUID: 525d9c62-1b87-5577-f76d-6f6d7bb4ba34
  Node UUID: 52a61733-bcf1-4c92-6436-001b2193b9a4
  Member UUIDs: ["52785959-6bee-21e0-e664-eca86bf99b3f", "52a61831-6060-22d7-c23d-001b2193b3b0", "52a61733-bcf1-4c92-6436-001b2193b9a4", "52aaffa8-d345-3a75-6809-005056bb3032"] (4)
Storage info:
  Auto claim: no
  Disk Mappings:
    SSD: SanDisk_SDSSDP064G - 59 GB
    MD: WDC_WD3000HLFS - 279 GB
    MD: WDC_WD3000HLFS - 279 GB
    MD: WDC_WD3000HLFS - 279 GB
NetworkInfo:
  Adapter: vmk1 (192.168.222.21)

vsan.cluster_info
Prints VSAN related information about all ESXi hosts. This command provides the same information as vsan.host_info:

  • Cluster role (master, backup or agent)
  • Cluster UUID
  • Node UUID
  • Member UUIDs
  • Auto claim (yes or no)
  • Disk Mapptings: Disks that are claimed by VSAN
  • NetworkInfo: VSAN traffic activated vmk adapters
usage: cluster_info cluster
Print VSAN info about a cluster
  cluster: Path to a ClusterComputeResource
  --help, -h: Show this message

Example 1 - Print VSAN related information from all host in a VSAN enabled Cluster:

/localhost/DC> vsan.cluster_info ~cluster
Host: esx1.virten.local
  VSAN enabled: yes
  Cluster info:
    Cluster role: agent
    Cluster UUID: 525d9c62-1b87-5577-f76d-6f6d7bb4ba34
    Node UUID: 52a61733-bcf1-4c92-6436-001b2193b9a4
    Member UUIDs: ["52785959-6bee-21e0-e664-eca86bf99b3f", "52a61831-6060-22d7-c23d-001b2193b3b0", "52a61733-bcf1-4c92-6436-001b2193b9a4", "52aaffa8-d345-3a75-6809-005056bb3032"] (4)
  Storage info:
    Auto claim: no
    Disk Mappings:
      SSD: SanDisk_SDSSDP064G - 59 GB
      MD: WDC_WD3000HLFS - 279 GB
      MD: WDC_WD3000HLFS - 279 GB
      MD: WDC_WD3000HLFS - 279 GB
  NetworkInfo:
    Adapter: vmk1 (192.168.222.21)

Host: esx2.virten.local
  VSAN enabled: yes
  Cluster info:
    Cluster role: backup
    Cluster UUID: 525d9c62-1b87-5577-f76d-6f6d7bb4ba34
    Node UUID: 52a61831-6060-22d7-c23d-001b2193b3b0
    Member UUIDs: ["52785959-6bee-21e0-e664-eca86bf99b3f", "52a61831-6060-22d7-c23d-001b2193b3b0", "52a61733-bcf1-4c92-6436-001b2193b9a4", "52aaffa8-d345-3a75-6809-005056bb3032"] (4)
  Storage info:
    Auto claim: no
    Disk Mappings:
      SSD: SanDisk_SDSSDP064G - 59 GB
      MD: WDC_WD3000HLFS - 279 GB
      MD: WDC_WD3000HLFS - 279 GB
      MD: WDC_WD3000HLFS - 279 GB
  NetworkInfo:
    Adapter: vmk1 (192.168.222.22)
[...]

vsan.check_limits
Gathers and checks various VSAN related counters like components or disk utilization against their limits. This command can be used against a single ESXi host or a Cluster.

usage: check_limits hosts_and_clusters...
Gathers (and checks) counters against limits
  hosts_and_clusters: Path to a HostSystem or ClusterComputeResource
  --help, -h: Show this message

Example 1 - Check VSAN limits from from all host in a VSAN enabled Cluster:

/localhost/DC> vsan.check_limits ~cluster
2013-12-26 14:21:02 +0000: Gathering stats from all hosts ...
2013-12-26 14:21:05 +0000: Gathering disks info ...
+-------------------+-------------------+------------------------+
| Host              | RDT               | Disks                  |
+-------------------+-------------------+------------------------+
| esx1.virten.local | Assocs: 51/20000  | Components: 45/750     |
|                   | Sockets: 26/10000 | WDC_WD3000HLFS: 44%    |
|                   | Clients: 4        | WDC_WD3000HLFS: 32%    |
|                   | Owners: 11        | WDC_WD3000HLFS: 28%    |
|                   |                   | SanDisk_SDSSDP064G: 0% |
| esx2.virten.local | Assocs: 72/20000  | Components: 45/750     |
|                   | Sockets: 24/10000 | WDC_WD3000HLFS: 29%    |
|                   | Clients: 5        | WDC_WD3000HLFS: 31%    |
|                   | Owners: 12        | WDC_WD3000HLFS: 43%    |
|                   |                   | SanDisk_SDSSDP064G: 0% |
| esx3.virten.local | Assocs: 88/20000  | Components: 45/750     |
|                   | Sockets: 31/10000 | WDC_WD3000HLFS: 42%    |
|                   | Clients: 6        | WDC_WD3000HLFS: 44%    |
|                   | Owners: 9         | WDC_WD3000HLFS: 38%    |
|                   |                   | SanDisk_SDSSDP064G: 0% |
+-------------------+-------------------+------------------------+

vsan.whatif_host_failures
Simulates how host failures would impact VSAN resource usage. The command shows the current VSAN disk usage and the calculated disk usage after a host has failed. The simulation assumes that all objects would be brought back to full policy compliance by bringing up new mirrors of existing data.

usage: whatif_host_failures [opts] hosts_and_clusters...
Simulates how host failures impact VSAN resource usage

The command shows current VSAN disk usage, but also simulates how
disk usage would evolve under a host failure. Concretely the simulation
assumes that all objects would be brought back to full policy
compliance by bringing up new mirrors of existing data.
The command makes some simplifying assumptions about disk space
balance in the cluster. It is mostly intended to do a rough estimate
if a host failure would drive the cluster to being close to full.
  hosts_and_clusters: Path to a HostSystem or ClusterComputeResource
  --num-host-failures-to-simulate, -n <i>: Number of host failures to simulate (default: 1)
        --show-current-usage-per-host, -s: Show current resources used per host
                               --help, -h: Show this message

Example 1 - Simulate 1 host failure:

/localhost/DC> vsan.whatif_host_failures ~cluster
Simulating 1 host failures:

+-----------------+-----------------------------+-----------------------------------+
| Resource        | Usage right now             | Usage after failure/re-protection |
+-----------------+-----------------------------+-----------------------------------+
| HDD capacity    |   7% used (1128.55 GB free) |  15% used (477.05 GB free)        |
| Components      |   2% used (2025 available)  |   3% used (1275 available)        |
| RC reservations |   0% used (90.47 GB free)   |   0% used (48.73 GB free)         |
+-----------------+-----------------------------+-----------------------------------+

Example 2 - Show utilization and simulate 1 host failure:

/localhost/DC> vsan.whatif_host_failures -s ~cluster
Current utilization of hosts:
+------------+---------+--------------+------+----------+----------------+--------------+
|            |         | HDD Capacity |      |          | Components     | SSD Capacity |
| Host       | NumHDDs | Total        | Used | Reserved | Used           | Reserved     |
+------------+---------+--------------+------+----------+----------------+--------------+
| 10.0.0.1   | 2       | 299.50 GB    | 6 %  | 5 %      |    4/562 (1 %) | 0 %          |
| 10.0.0.2   | 2       | 299.50 GB    | 10 % | 9 %      |   11/562 (2 %) | 0 %          |
| 10.0.0.3   | 2       | 299.50 GB    | 10 % | 9 %      |    6/562 (1 %) | 0 %          |
| 10.0.0.4   | 2       | 299.50 GB    | 14 % | 13 %     |    7/562 (1 %) | 0 %          |
+------------+---------+--------------+------+----------+----------------+--------------+

Simulating 1 host failures:

+-----------------+-----------------------------+-----------------------------------+
| Resource        | Usage right now             | Usage after failure/re-protection |
+-----------------+-----------------------------+-----------------------------------+
| HDD capacity    |  10% used (1079.73 GB free) |  13% used (780.23 GB free)        |
| Components      |   1% used (2220 available)  |   2% used (1658 available)        |
| RC reservations |   0% used (55.99 GB free)   |   0% used (41.99 GB free)         |
+-----------------+-----------------------------+-----------------------------------+

vsan.enter_maintenance_mode
Put the host into maintenance mode. This command is VSAN aware and can migrate VSAN data to other host like the vSphere Web Client does. It also migrates running virtual machines when DRS is enabled.

usage: enter_maintenance_mode [opts] host...
Put hosts into maintenance mode
  host: Path to a HostSystem
               --timeout, -t <i>: Timeout (default: 0)
  --evacuate-powered-off-vms, -e: Evacuate powered off vms
                   --no-wait, -n: Don't wait for Task to complete
             --vsan-mode, -v <s>: Actions to take for VSAN backed 
                                  storage (default: ensureObjectAccessibility)
                      --help, -h: Show this message

Further explanation of the 4 options:

  • timeout, -t: Set a timoute for the process to complete. When the host can not enter maintenance mode in X seconds, the process is canceled.
  • evacuate-powered-off-vms, -e: Moves powered off virtual machines to other hosts in the cluster.
  • no-wait, -n: The command returns immediately without waiting for the task to complete.
  • vsan-mode, -v: Actions to take for VSAN components. Options:
    • ensureObjectAccessibility (default)
    • evacuateAllData
    • noAction

Example 1 - Put the host into maintenance mode. Do not copy any VSAN components (Fast but with reduced redundancy):

/localhost/DC> vsan.enter_maintenance_mode ~esx
EnterMaintenanceMode esx1.virten.local: success

Example 2 - Put the host into maintenance mode. Copy all VSAN components components to other hosts in the cluster:

/localhost/DC> vsan.enter_maintenance_mode ~esx -v evacuateAllData
EnterMaintenanceMode esx1.virten.local: success

Example 3 - Put the host into maintenance mode. Copy all VSAN components components to other hosts in the cluster. Cancel the process when it takes longer than 10 minutes:

/localhost/DC> vsan.enter_maintenance_mode ~esx -v evacuateAllData -t 600
EnterMaintenanceMode esx1.virten.local: success

Example 4 - Put the host into maintenance mode. Do not track the process (Batch mode) :

/localhost/DC> vsan.enter_maintenance_mode ~esx -n
/localhost/DC>

vsan.resync_dashboard
This command shows what happens when a mirror resync is in progress. If a host fails or is going into maintenance mode, you should watch the resync status here. The command can be run once or with an refresh interval.

usage: resync_dashboard [opts] cluster_or_host
Resyncing dashboard
   cluster_or_host: Path to a ClusterComputeResource or HostSystem
   --refresh-rate, -r <i>: Refresh interval (in sec). Default is no refresh
               --help, -h: Show this message

Example 1 - Resync Dashboard where nothing happens:

/localhost/DC>  vsan.resync_dashboard ~cluster
2013-12-27 20:05:56 +0000: Querying all VMs on VSAN ...
2013-12-27 20:05:56 +0000: Querying all objects in the system from esx1.virten.lab ...
2013-12-27 20:05:56 +0000: Got all the info, computing table ...
+-----------+-----------------+---------------+
| VM/Object | Syncing objects | Bytes to sync |
+-----------+-----------------+---------------+
+-----------+-----------------+---------------+
| Total     | 0               | 0.00 GB       |
+-----------+-----------------+---------------+

Example 2 - Resync Dashboerd after a host has been put into maintenance mode. Refreshed every 10 seconds:

/localhost/DC>  vsan.resync_dashboard ~cluster --refresh-rate 10
2013-12-27 20:39:46 +0000: Querying all VMs on VSAN ...
2013-12-27 20:39:46 +0000: Querying all objects in the system from esx1.virten.local ...
2013-12-27 20:39:47 +0000: Got all the info, computing table ...
+-----------+-----------------+---------------+
| VM/Object | Syncing objects | Bytes to sync |
+-----------+-----------------+---------------+
+-----------+-----------------+---------------+
| Total     | 0               | 0.00 GB       |
+-----------+-----------------+---------------+
2013-12-27 20:39:57 +0000: Querying all objects in the system from esx1.virten.local ...
2013-12-27 20:39:59 +0000: Got all the info, computing table ...
+-----------------------------------------------------------------+-----------------+---------------+
| VM/Object                                                       | Syncing objects | Bytes to sync |
+-----------------------------------------------------------------+-----------------+---------------+
| vm1                                                             | 1               |               |
|    [vsanDatastore] 5078bd52-2977-8cf9-107c-00505687439c/vm1.vmx |                 | 0.17 GB       |
+-----------------------------------------------------------------+-----------------+---------------+
| Total                                                           | 1               | 0.17 GB       |
+-----------------------------------------------------------------+-----------------+---------------+
2013-12-27 20:40:14 +0000: Querying all objects in the system from esx1.virten.local ...
2013-12-27 20:40:17 +0000: Got all the info, computing table ...
+--------------------------------------------------------------------+-----------------+---------------+
| VM/Object                                                          | Syncing objects | Bytes to sync |
+--------------------------------------------------------------------+-----------------+---------------+
| vm1                                                                | 1               |               |
|    [vsanDatastore] 5078bd52-2977-8cf9-107c-00505687439c/vm1.vmx    |                 | 0.34 GB       |
| debian                                                             | 1               |               |
|    [vsanDatastore] 6978bd52-4d92-05ed-dad2-005056871792/debian.vmx |                 | 0.35 GB       |
+--------------------------------------------------------------------+-----------------+---------------+
| Total                                                              | 2               | 0.69 GB       |
+--------------------------------------------------------------------+-----------------+---------------+
[...]

Object Management

vsan.disks_info
Prints informations about all physical disks from a host. This command helps to identify backend disks and their state. It also provides hints on why physical disks are ineligible for VSAN. This command retains information about:

  • Disk Display Name
  • Disk type (SSD or MD)
  • RAW Size
  • State (inUse, eligible or ineligible)
  • Host Adapter (only with -s option)
  • Existing partitions
usage: disks_info [opts] host...
Print physical disk info about a host
  host: Path to a HostSystem
  --show-adapters, -s: Show adapter information
           --help, -h: Show this message

Example 1 - Print host disk information

/localhost/DC> vsan.disks_info ~esx
Disks on host esx1.virten.local:
+-------------------------------------+-------+--------+----------------------------------------+
| DisplayName                         | isSSD | Size   | State                                  |
+-------------------------------------+-------+--------+----------------------------------------+
| WDC_WD3000HLFS                      | MD    | 279 GB | inUse                                  |
| ATA WDC WD3000HLFS-0                |       |        |                                        |
+-------------------------------------+-------+--------+----------------------------------------+
| WDC_WD3000HLFS                      | MD    | 279 GB | inUse                                  |
| ATA WDC WD3000HLFS-0                |       |        |                                        |
+-------------------------------------+-------+--------+----------------------------------------+
| WDC_WD3000HLFS                      | MD    | 279 GB | inUse                                  |
| ATA WDC WD3000HLFS-0                |       |        |                                        |
+-------------------------------------+-------+--------+----------------------------------------+
| SanDisk_SDSSDP064G                  | SSD   | 59 GB  | inUse                                  |
| ATA SanDisk SDSSDP06                |       |        |                                        |
+-------------------------------------+-------+--------+----------------------------------------+
| Local USB  (mpx.vmhba32:C0:T0:L0)   | MD    | 7 GB   | ineligible (Existing partitions found  |
|  USB DISK 2.0                       |       |        | on disk 'mpx.vmhba32:C0:T0:L0'.)       |
|                                     |       |        |                                        |
|                                     |       |        | Partition table:                       |
|                                     |       |        | 5: 0.24 GB, type = vfat                |
|                                     |       |        | 6: 0.24 GB, type = vfat                |
|                                     |       |        | 7: 0.11 GB, type = coredump            |
|                                     |       |        | 8: 0.28 GB, type = vfat                |
+-------------------------------------+-------+--------+----------------------------------------+

vsan.disks_stats
Prints stats on all disks in VSAN. Can be used against a host or the whole cluster. When used against a host, it does only resolves names from the given host. I would suggest to run this command against clusters only. That displays names for all disks and hosts in the VSAN. It is very helpfull when you are troubleshooting disk full errors. The command retains information about:

  • Disk Display Name
  • Disk Size
  • Disk type (SSD or MD)
  • Number of Components
  • Capacity
  • Used/Reserved percentage
  • Health Status
usage: disks_stats [opts] hosts_and_clusters...
Show stats on all disks in VSAN
  hosts_and_clusters: Path to a HostSystem or ClusterComputeResource
  --compute-number-of-components, -c: Deprecated
                     --show-iops, -s: Show deprecated fields
                          --help, -h: Show this message

The -c option is deprecated, the number of components is displayed by default. The value displayed with the -s option is an assumption about the drives maximum possible iops.

Example 1 - Print VSAN Disk stats from all hosts in a VSAN enabled Cluster:

/localhost/DC> vsan.disks_stats ~cluster
+---------------------------------+-----------------+-------+------+-----------+------+----------+--------+
|                                 |                 |       | Num  | Capacity  |      |          | Status |
| DisplayName                     | Host            | isSSD | Comp | Total     | Used | Reserved | Health |
+---------------------------------+-----------------+-------+------+-----------+------+----------+--------+
| t10.ATA_____SanDisk_SDSSDP064G  | esx1.virten.loc | SSD   | 0    | 41.74 GB  | 0 %  | 0 %      | OK     |
| t10.ATA_____WDC_WD3000HLFS      | esx1.virten.loc | MD    | 30   | 279.25 GB | 43 % | 42 %     | OK     |
| t10.ATA_____WDC_WD3000HLFS      | esx1.virten.loc | MD    | 32   | 279.25 GB | 41 % | 40 %     | OK     |
| t10.ATA_____WDC_WD3000HLFS      | esx1.virten.loc | MD    | 24   | 279.25 GB | 33 % | 32 %     | OK     |
+---------------------------------+-----------------+-------+------+-----------+------+----------+--------+
| t10.ATA_____SanDisk_SDSSDP064G  | esx2.virten.loc | SSD   | 0    | 41.74 GB  | 0 %  | 0 %      | OK     |
| t10.ATA_____WDC_WD3000HLFS      | esx2.virten.loc | MD    | 32   | 279.25 GB | 44 % | 44 %     | OK     |
| t10.ATA_____WDC_WD3000HLFS      | esx2.virten.loc | MD    | 34   | 279.25 GB | 46 % | 44 %     | OK     |
| t10.ATA_____WDC_WD3000HLFS      | esx2.virten.loc | MD    | 28   | 279.25 GB | 33 % | 32 %     | OK     |
+---------------------------------+-----------------+-------+------+-----------+------+----------+--------+
| t10.ATA_____SanDisk_SDSSDP064G  | esx3.virten.loc | SSD   | 0    | 41.74 GB  | 0 %  | 0 %      | OK     |
| t10.ATA_____WDC_WD3000HLFS      | esx3.virten.loc | MD    | 35   | 279.25 GB | 42 % | 42 %     | OK     |
| t10.ATA_____WDC_WD3000HLFS      | esx3.virten.loc | MD    | 33   | 279.25 GB | 43 % | 42 %     | OK     |
| t10.ATA_____WDC_WD3000HLFS      | esx3.virten.loc | MD    | 22   | 279.25 GB | 36 % | 34 %     | OK     |
+---------------------------------+-----------------+-------+------+-----------+------+----------+--------+

vsan.vm_object_info
Prints VSAN object information about a VM. This command is the equivalent to the Manage > VM Storage Policies tab in the vSphere Web Client and allows you to idenify where stripes, mirrors and witness of virtual disks are located. The command retains information about:

  • Namespace directory (Virtual Machine home directory)
  • Disk backing (Virtual Disks)
  • Number of objects (DOM Objects)
  • UUID from objects and components (usful for other commands)
  • Location of object stripes and mirrors
  • Location of object witness
  • Storage Policy (hostFailuresToTolerate, forceProvisioning, stripeWidth, etc.)
  • Resync Status
usage: vm_object_info [opts] vms...
Fetch VSAN object information about a VM
  vms: Path to a VirtualMachine
                --cluster, -c <s>: Cluster on which to fetch the object info
  --perspective-from-host, -p <s>: Host to query object info from
     --include-detailed-usage, -i: Include detailed usage info
                       --help, -h: Show this message

The -i parameter displays additional information about the physical usage (physUsage) from objects.

Example 1 - Print VM object information:

/localhost/DC> vsan.vm_object_info ~vm
VM vma.virten.lab:
  Namespace directory
    DOM Object: f3bfa952-175f-4d7e-1eae-001b2193b3b0 (owner: esx2.virten.lab, policy: hostFailuresToTolerate = 1, stripeWidth = 1, spbmProfileId = fbd74e7a-2bf9-481d-88b6-22c0abbc8898, proportionalCapacity = [0, 100], spbmProfileGenerationNumber = 1)
      Witness: eb16af52-dd3f-b6c4-19f6-001b2193b3b0 (state: ACTIVE (5), host: esx1.virten.lab, md: t10.ATA_____WD3000HLFS, ssd: t10.ATA_____SanDisk_SDSSDP064G)
      Witness: eb16af52-1865-b3c4-3abd-001b2193b3b0 (state: ACTIVE (5), host: esx2.virten.lab, md: t10.ATA_____WD3000HLFS, ssd: t10.ATA_____SanDisk_SDSSDP064G)
      Witness: eb16af52-805a-b5c4-cf5d-001b2193b3b0 (state: ACTIVE (5), host: esx1.virten.lab, md: t10.ATA_____WD3000HLFS, ssd: t10.ATA_____SanDisk_SDSSDP064G)
      RAID_1
        RAID_0
          Component: 2c16af52-89fa-db48-09f8-001b2193b3b0 (state: ACTIVE (5), host: esx4.virten.lab, md: t10.ATA_____WD3000HLFS, ssd: t10.ATA_____SanDisk_SDSSDP064G)
          Component: 2c16af52-6f1f-da48-5c92-001b2193b3b0 (state: ACTIVE (5), host: esx4.virten.lab, md: t10.ATA_____WD3000HLFS, ssd: t10.ATA_____SanDisk_SDSSDP064G)
        RAID_0
          Component: f3bfa952-5d7b-f4ba-190f-001b2193b3b0 (state: ACTIVE (5), host: esx2.virten.lab, md: t10.ATA_____WD3000HLFS, ssd: t10.ATA_____SanDisk_SDSSDP064G)
          Component: f3bfa952-958b-f3ba-5648-001b2193b3b0 (state: ACTIVE (5), host: esx1.virten.lab, md: t10.ATA_____WD3000HLFS, ssd: t10.ATA_____SanDisk_SDSSDP064G)
  Disk backing: [vsanDatastore] f3bfa952-175f-4d7e-1eae-001b2193b3b0/vma.virten.lab.vmdk
    DOM Object: fabfa952-721d-fc1f-82d3-001b2193b3b0 (owner: esx2.virten.lab, policy: spbmProfileGenerationNumber = 1, hostFailuresToTolerate = 1, spbmProfileId = fbd74e7a-2bf9-481d-88b6-22c0abbc8898)
      Witness: e916af52-075e-9740-f1c7-001b2193b3b0 (state: ACTIVE (5), host: esx4.virten.lab, md: t10.ATA_____WD3000HLFS, ssd: t10.ATA_____SanDisk_SDSSDP064G)
      RAID_1
        Component: f015af52-7c95-0d57-d7b3-001b2193b3b0 (state: ACTIVE (5), host: esx1.virten.local, md: t10.ATA_____WD3000HLFS, ssd: t10.ATA_____SanDisk_SDSSDP064G, dataToSync: 0.00 GB)
        Component: fabfa952-aa57-d02c-9204-001b2193b3b0 (state: ACTIVE (5), host: esx2.virten.lab, md: t10.ATA_____WD3000HLFS, ssd: t10.ATA_____SanDisk_SDSSDP064G)

What we see here is a virtual machine with one virtual disk. That are 2 DOM Objects - The Namespace directory and the virtual disk.
vsan-physical-disk-placement

 

vsan.cmmds_find
A powerful command to find objects and detailed object information. Can be used against hosts or clusters. Usage against clusters is recommended to resolve UUID into readable names.

usage: cmmds_find [opts] cluster_or_host
CMMDS Find
  cluster_or_host: Path to a ClusterComputeResource or HostSystem
   --type, -t <s>: CMMDS type, e.g. DOM_OBJECT, LSOM_OBJECT, POLICY, DISK etc.
   --uuid, -u <s>: UUID of the entry.
  --owner, -o <s>: UUID of the owning node.
       --help, -h: Show this message

Example 1 - List all Disks in VSAN:

/localhost/DC> vsan.cmmds_find ~cluster -t DISK
+----+------+--------------------------------------+-----------------+---------+-------------------------------------------------------+
| #  | Type | UUID                                 | Owner           | Health  | Content                                               |
+----+------+--------------------------------------+-----------------+---------+-------------------------------------------------------+
| 1  | DISK | 527547f8-247e-9901-c532-04b8d5b952b0 | esx2.virten.lab | Healthy | {"capacity"=>299842404352,                            |
|    |      |                                      |                 |         |  "iops"=>100,                                         |
|    |      |                                      |                 |         |  "iopsWritePenalty"=>10000000,                        |
|    |      |                                      |                 |         |  "throughput"=>200000000,                             |
|    |      |                                      |                 |         |  "throughputWritePenalty"=>0,                         |
|    |      |                                      |                 |         |  "latency"=>3400000,                                  |
|    |      |                                      |                 |         |  "latencyDeviation"=>0,                               |
|    |      |                                      |                 |         |  "reliabilityBase"=>10,                               |
|    |      |                                      |                 |         |  "reliabilityExponent"=>15,                           |
|    |      |                                      |                 |         |  "mtbf"=>1600000,                                     |
|    |      |                                      |                 |         |  "l2CacheCapacity"=>0,                                |
|    |      |                                      |                 |         |  "l1CacheCapacity"=>16777216,                         |
|    |      |                                      |                 |         |  "isSsd"=>0,                                          |
|    |      |                                      |                 |         |  "ssdUuid"=>"526209e5-118a-5fbd-e1d0-0bdd816b32eb",   |
|    |      |                                      |                 |         |  "volumeName"=>"52a9bf13-6b13325f-e4da-001b2193b3b0"} |
| 2  | DISK | 526e0510-6461-3410-5be0-0d9ec1b1495f | esx4.virten.lab | Healthy | {"capacity"=>53418655744,                             |
|    |      |                                      |                 |         |  "iops"=>100,                                         |
|    |      |                                      |                 |         |  "iopsWritePenalty"=>10000000,                        |
|    |      |                                      |                 |         |  "throughput"=>200000000,                             |
|    |      |                                      |                 |         |  "throughputWritePenalty"=>0,                         |
|    |      |                                      |                 |         |  "latency"=>3400000,                                  |
|    |      |                                      |                 |         |  "latencyDeviation"=>0,                               |
|    |      |                                      |                 |         |  "reliabilityBase"=>10,                               |
|    |      |                                      |                 |         |  "reliabilityExponent"=>15,                           |
|    |      |                                      |                 |         |  "mtbf"=>1600000,                                     |
|    |      |                                      |                 |         |  "l2CacheCapacity"=>0,                                |
|    |      |                                      |                 |         |  "l1CacheCapacity"=>16777216,                         |
|    |      |                                      |                 |         |  "isSsd"=>0,                                          |
|    |      |                                      |                 |         |  "ssdUuid"=>"520f544a-1d27-aecc-cbf4-2f170e4bf0f8",   |
|    |      |                                      |                 |         |  "volumeName"=>"52af1537-3d763738-2c93-005056bb3032"} |
[...]

Example 2 - List all disks from a specific ESXi Host. Identify the hosts UUID (Node UUID) with vsan.host_info:

/localhost/DC> vsan.host_info ~esx
VSAN enabled: yes
Cluster info:
  Cluster role: agent
  Cluster UUID: 525d9c62-1b87-5577-f76d-6f6d7bb4ba34
  Node UUID: 52a61733-bcf1-4c92-6436-001b2193b9a4
  Member UUIDs: ["52785959-6bee-21e0-e664-eca86bf99b3f", "52a61831-6060-22d7-c23d-001b2193b3b0", "52a61733-bcf1-4c92-6436-001b2193b9a4", "52aaffa8-d345-3a75-6809-005056bb3032"] (4)
Storage info:
  Auto claim: no
  Disk Mappings:
    SSD: SanDisk_SDSSDP064G - 59 GB
    MD: WDC_WD1500HLFS2D01G6U3 - 139 GB
    MD: WDC_WD1500ADFD2D00NLR1 - 139 GB
    MD: VB0250EAVER - 232 GB
NetworkInfo:
  Adapter: vmk1 (192.168.222.21)
/localhost/DC> vsan.cmmds_find ~cluster -t DISK -o 52a61733-bcf1-4c92-6436-001b2193b9a4
+---+------+--------------------------------------+-----------------+---------+-------------------------------------------------------+
| # | Type | UUID                                 | Owner           | Health  | Content                                               |
+---+------+--------------------------------------+-----------------+---------+-------------------------------------------------------+
| 1 | DISK | 52f853e4-7eca-2e9c-b157-843aefcde6dd | esx1.virten.lab | Healthy | {"capacity"=>149786984448,                            |
|   |      |                                      |                 |         |  "iops"=>100,                                         |
|   |      |                                      |                 |         |  "iopsWritePenalty"=>10000000,                        |
|   |      |                                      |                 |         |  "throughput"=>200000000,                             |
|   |      |                                      |                 |         |  "throughputWritePenalty"=>0,                         |
|   |      |                                      |                 |         |  "latency"=>3400000,                                  |
|   |      |                                      |                 |         |  "latencyDeviation"=>0,                               |
|   |      |                                      |                 |         |  "reliabilityBase"=>10,                               |
|   |      |                                      |                 |         |  "reliabilityExponent"=>15,                           |
|   |      |                                      |                 |         |  "mtbf"=>1600000,                                     |
|   |      |                                      |                 |         |  "l2CacheCapacity"=>0,                                |
|   |      |                                      |                 |         |  "l1CacheCapacity"=>16777216,                         |
|   |      |                                      |                 |         |  "isSsd"=>0,                                          |
|   |      |                                      |                 |         |  "ssdUuid"=>"5292ca0e-5315-c0a8-6396-bd7d93a92d92",   |
|   |      |                                      |                 |         |  "volumeName"=>"52a9befe-6a2ccbfe-9d16-001b2193b9a4"} |
| 2 | DISK | 5292ca0e-5315-c0a8-6396-bd7d93a92d92 | esx1.virten.lab | Healthy | {"capacity"=>44812992512,                             |
|   |      |                                      |                 |         |  "iops"=>20000,                                       |
|   |      |                                      |                 |         |  "iopsWritePenalty"=>10000000,                        |
|   |      |                                      |                 |         |  "throughput"=>200000000,                             |
|   |      |                                      |                 |         |  "throughputWritePenalty"=>0,                         |
|   |      |                                      |                 |         |  "latency"=>3400000,                                  |
|   |      |                                      |                 |         |  "latencyDeviation"=>0,                               |
|   |      |                                      |                 |         |  "reliabilityBase"=>10,                               |
|   |      |                                      |                 |         |  "reliabilityExponent"=>15,                           |
|   |      |                                      |                 |         |  "mtbf"=>2000000,                                     |
|   |      |                                      |                 |         |  "l2CacheCapacity"=>0,                                |
|   |      |                                      |                 |         |  "l1CacheCapacity"=>16777216,                         |
|   |      |                                      |                 |         |  "isSsd"=>1,                                          |
|   |      |                                      |                 |         |  "ssdUuid"=>"5292ca0e-5315-c0a8-6396-bd7d93a92d92",   |
|   |      |                                      |                 |         |  "volumeName"=>"NA"}                                  |
| 3 | DISK | 5287fb1c-4cfc-04e0-6a22-281b4c259a1c | esx1.virten.lab | Healthy | {"capacity"=>249913409536,                            |
|   |      |                                      |                 |         |  "iops"=>100,                                         |
|   |      |                                      |                 |         |  "iopsWritePenalty"=>10000000,                        |
[...]

Example 3 - List DOM Objects from a specific ESXi Host:

/localhost/DC> vsan.cmmds_find ~cluster -t DOM_OBJECT -o 52a61733-bcf1-4c92-6436-001b2193b9a4
[...]
| 16 | DOM_OBJECT | 7e62c152-7dfb-c6e5-07b8-001b2193b9a4 | esx1.virten.lab | Healthy | {"type"=>"Configuration",                                          |
|    |            |                                      |                 |         |  "attributes"=>                                                    |
|    |            |                                      |                 |         |   {"CSN"=>3,                                                       |
|    |            |                                      |                 |         |    "addressSpace"=>268435456,                                      |
|    |            |                                      |                 |         |    "compositeUuid"=>"7e62c152-7dfb-c6e5-07b8-001b2193b9a4"},       |
|    |            |                                      |                 |         |  "child-1"=>                                                       |
|    |            |                                      |                 |         |   {"type"=>"RAID_1",                                               |
|    |            |                                      |                 |         |    "attributes"=>{},                                               |
|    |            |                                      |                 |         |    "child-1"=>                                                     |
|    |            |                                      |                 |         |     {"type"=>"Component",                                          |
|    |            |                                      |                 |         |      "attributes"=>                                                |
|    |            |                                      |                 |         |       {"capacity"=>268435456,                                      |
|    |            |                                      |                 |         |        "addressSpace"=>268435456,                                  |
|    |            |                                      |                 |         |        "componentState"=>5,                                        |
|    |            |                                      |                 |         |        "componentStateTS"=>1388405374,                             |
|    |            |                                      |                 |         |        "faultDomainId"=>"52a61831-6060-22d7-c23d-001b2193b3b0"},   |
|    |            |                                      |                 |         |      "componentUuid"=>"7e62c152-763d-1400-2b06-001b2193b9a4",      |
|    |            |                                      |                 |         |      "diskUuid"=>"527547f8-247e-9901-c532-04b8d5b952b0"},          |
|    |            |                                      |                 |         |    "child-2"=>                                                     |
|    |            |                                      |                 |         |     {"type"=>"Component",                                          |
|    |            |                                      |                 |         |      "attributes"=>                                                |
|    |            |                                      |                 |         |       {"capacity"=>268435456,                                      |
|    |            |                                      |                 |         |        "addressSpace"=>268435456,                                  |
|    |            |                                      |                 |         |        "componentState"=>5,                                        |
|    |            |                                      |                 |         |        "componentStateTS"=>1388590529,                             |
|    |            |                                      |                 |         |        "staleLsn"=>0,                                              |
|    |            |                                      |                 |         |        "faultDomainId"=>"52a61733-bcf1-4c92-6436-001b2193b9a4"},   |
|    |            |                                      |                 |         |      "componentUuid"=>"c135c452-2f04-0533-dbbc-001b2193b9a4",      |
|    |            |                                      |                 |         |      "diskUuid"=>"52f853e4-7eca-2e9c-b157-843aefcde6dd"}},         |
|    |            |                                      |                 |         |  "child-2"=>                                                       |
|    |            |                                      |                 |         |   {"type"=>"Witness",                                              |
|    |            |                                      |                 |         |    "attributes"=>                                                  |
|    |            |                                      |                 |         |     {"componentState"=>5,                                          |
|    |            |                                      |                 |         |      "isWitness"=>1,                                               |
|    |            |                                      |                 |         |      "faultDomainId"=>"52aaffa8-d345-3a75-6809-005056bb3032"},     |
|    |            |                                      |                 |         |    "componentUuid"=>"c135c452-cd77-0733-1708-001b2193b9a4",        |
|    |            |                                      |                 |         |    "diskUuid"=>"526e0510-6461-3410-5be0-0d9ec1b1495f"}}            |
[...]

Example 4 - List LSOM Objects (Components) from a specific ESXi Host:

/localhost/DC> vsan.cmmds_find ~cluster -t LSOM_OBJECT -o 52a61733-bcf1-4c92-6436-001b2193b9a4
[...]
| 95  | LSOM_OBJECT | 2739c452-a2cf-e9ed-9d07-eca86bf99b3f | esx1.virten.lab | Healthy | {"diskUuid"=>"52bc69ac-35ad-c2f7-bab8-2ecf298cd4e5",      |
|     |             |                                      |                 |         |  "compositeUuid"=>"1d61c152-d60c-6a95-5470-eca86bf99b3f", |
|     |             |                                      |                 |         |  "capacityUsed"=>17181966336,                             |
|     |             |                                      |                 |         |  "physCapacityUsed"=>0}                                   |
| 96  | LSOM_OBJECT | 895fc152-391f-c4f2-a697-eca86bf99b3f | esx1.virten.lab | Healthy | {"diskUuid"=>"52bc69ac-35ad-c2f7-bab8-2ecf298cd4e5",      |
|     |             |                                      |                 |         |  "compositeUuid"=>"895fc152-3056-1cde-3a1b-eca86bf99b3f", |
|     |             |                                      |                 |         |  "capacityUsed"=>53477376,                                |
|     |             |                                      |                 |         |  "physCapacityUsed"=>53477376}                            |
| 97  | LSOM_OBJECT | 6537c452-a9b1-f4f5-badb-001b2193b3b0 | esx1.virten.lab | Healthy | {"diskUuid"=>"52f853e4-7eca-2e9c-b157-843aefcde6dd",      |
|     |             |                                      |                 |         |  "compositeUuid"=>"ff5fc152-65b6-bc34-ff54-001b2193b9a4", |
|     |             |                                      |                 |         |  "capacityUsed"=>2097152,                                 |
|     |             |                                      |                 |         |  "physCapacityUsed"=>0}                                   |
[...]

vsan.disk_object_info
Prints all objects that are located on a physical disk. This command helps during troubleshooting when you want to identify all objects on a physical disk. You have to know the disk UUID which can be identified with the vsan.cmmds_find command.

usage: disk_object_info cluster_or_host disk_uuid...
Fetch information about all VSAN objects on a given physical disk
  cluster_or_host: Cluster or host on which to fetch the object info ClusterComputeResource or HostSystem
  disk_uuid:
  --help, -h: Show this message

Example 1 - Get Disk UUID with vsan.cmmds_find and display all objects located on this disk:

/localhost/DC> vsan.cmmds_find ~cluster -t DISK
+---+------+--------------------------------------+-----------------+---------+-------------------------------------------------------+
| # | Type | UUID                                 | Owner           | Health  | Content                                               |
+---+------+--------------------------------------+-----------------+---------+-------------------------------------------------------+
| 1 | DISK | 52f853e4-7eca-2e9c-b157-843aefcde6dd | esx1.virten.lab | Healthy | {"capacity"=>149786984448,                            |
|   |      |                                      |                 |         |  "iops"=>100,                                         |
|   |      |                                      |                 |         |  "iopsWritePenalty"=>10000000,                        |
|   |      |                                      |                 |         |  "throughput"=>200000000,                             |
|   |      |                                      |                 |         |  "throughputWritePenalty"=>0,                         |
|   |      |                                      |                 |         |  "latency"=>3400000,                                  |
|   |      |                                      |                 |         |  "latencyDeviation"=>0,                               |
|   |      |                                      |                 |         |  "reliabilityBase"=>10,                               |
|   |      |                                      |                 |         |  "reliabilityExponent"=>15,                           |
|   |      |                                      |                 |         |  "mtbf"=>1600000,                                     |
|   |      |                                      |                 |         |  "l2CacheCapacity"=>0,                                |
|   |      |                                      |                 |         |  "l1CacheCapacity"=>16777216,                         |
|   |      |                                      |                 |         |  "isSsd"=>0,                                          |
|   |      |                                      |                 |         |  "ssdUuid"=>"5292ca0e-5315-c0a8-6396-bd7d93a92d92",   |
|   |      |                                      |                 |         |  "volumeName"=>"52a9befe-6a2ccbfe-9d16-001b2193b9a4"} |
[...]
/localhost/DC> vsan.disk_object_info ~cluster 52f853e4-7eca-2e9c-b157-843aefcde6dd
Fetching VSAN disk info from [...] (this may take a moment) ...
Physical disk mpx.vmhba1:C0:T4:L0 (52f853e4-7eca-2e9c-b157-843aefcde6dd):
  DOM Object: 27e4bd52-49f6-050e-178b-00505687439c (owner: esx1.virten.lab, policy: hostFailuresToTolerate = 1, stripeWidth = 2, forceProvisioning = 1, proportionalCapacity = 100)
    Context: Part of VM vm1: Disk: [vsanDatastore] 5078bd52-2977-8cf9-107c-00505687439c/vm1_1.vmdk
    Witness: bae5bd52-54f7-05ab-f4b3-00505687439c (state: ACTIVE (5), host: esx1.virten.lab, md: mpx.vmhba1:C0:T2:L0, ssd: mpx.vmhba1:C0:T1:L0)
    Witness: bae5bd52-ebf9-ffaa-cbb4-00505687439c (state: ACTIVE (5), host: esx1.virten.lab, md: mpx.vmhba1:C0:T4:L0, ssd: mpx.vmhba1:C0:T1:L0)
    Witness: bae5bd52-e59e-03ab-b080-00505687439c (state: ACTIVE (5), host: esx1.virten.lab, md: **mpx.vmhba1:C0:T4:L0**, ssd: mpx.vmhba1:C0:T1:L0)
    RAID_1
      RAID_0
        Component: abe5bd52-4128-cf28-ef6f-00505687439c (state: ACTIVE (5), host: esx2.virtenlab, md: mpx.vmhba1:C0:T4:L0, ssd: mpx.vmhba1:C0:T1:L0)
        Component: abe5bd52-1516-cb28-6e87-00505687439c (state: ACTIVE (5), host: esx2.virtenlab, md: mpx.vmhba1:C0:T2:L0, ssd: mpx.vmhba1:C0:T1:L0)
      RAID_0
        Component: 27e4bd52-4048-1465-ac76-00505687439c (state: ACTIVE (5), host: esx4.virtenlab, md: **mpx.vmhba1:C0:T4:L0**, ssd: mpx.vmhba1:C0:T1:L0)
        Component: 27e4bd52-1d88-1265-22ab-00505687439c (state: ACTIVE (5), host: esx4.virtenlab, md: mpx.vmhba1:C0:T2:L0, ssd: mpx.vmhba1:C0:T1:L0)
  DOM Object: 7e78bd52-7595-1716-85a2-005056871792 (owner: esx1.virtenlab, policy: hostFailuresToTolerate = 1, stripeWidth = 2, forceProvisioning = 1, proportionalCapacity = 100)
    Context: Part of VM debian: Disk: [vsanDatastore] 6978bd52-4d92-05ed-dad2-005056871792/debian.vmdk
    Witness: aee5bd52-2ae2-197b-e67a-005056871792 (state: ACTIVE (5), host: esx4.virtenlab, md: mpx.vmhba1:C0:T4:L0, ssd: mpx.vmhba1:C0:T1:L0)
    Witness: aee5bd52-8557-0d7b-3022-005056871792 (state: ACTIVE (5), host: esx2.virtenlab, md: mpx.vmhba1:C0:T4:L0, ssd: mpx.vmhba1:C0:T1:L0)
    Witness: aee5bd52-7443-177b-74a8-005056871792 (state: ACTIVE (5), host: esx2.virtenlab, md: mpx.vmhba1:C0:T4:L0, ssd: mpx.vmhba1:C0:T1:L0)
    RAID_1
      RAID_0
        Component: 36debd52-7390-a05d-9225-005056871792 (state: ACTIVE (5), host: esx4.virtenlab, md: mpx.vmhba1:C0:T2:L0, ssd: mpx.vmhba1:C0:T1:L0)
        Component: 36debd52-a9b8-965d-03a6-005056871792 (state: ACTIVE (5), host: esx4.virtenlab, md: **mpx.vmhba1:C0:T4:L0**, ssd: mpx.vmhba1:C0:T1:L0)
      RAID_0
        Component: 7f78bd52-2d59-c558-09f9-005056871792 (state: ACTIVE (5), host: esx3.virtenlab, md: mpx.vmhba1:C0:T2:L0, ssd: mpx.vmhba1:C0:T1:L0)
        Component: 7f78bd52-d827-c458-9d94-005056871792 (state: ACTIVE (5), host: esx3.virtenlab, md: mpx.vmhba1:C0:T4:L0, ssd: mpx.vmhba1:C0:T1:L0)
  DOM Object: 2ee4bd52-d1af-dd19-626d-00505687439c (owner: esx3.virtenlab, policy: hostFailuresToTolerate = 1, stripeWidth = 2, forceProvisioning = 1, proportionalCapacity = 100)
    Context: Part of VM vm1: Disk: [vsanDatastore] 5078bd52-2977-8cf9-107c-00505687439c/vm1_2.vmdk
    Witness: bde5bd52-959d-5d8b-6058-00505687439c (state: ACTIVE (5), host: esx3.virtenlab, md: **mpx.vmhba1:C0:T4:L0**, ssd: mpx.vmhba1:C0:T1:L0)
    Witness: bde5bd52-4a48-588b-e464-00505687439c (state: ACTIVE (5), host: esx1.virtenlab, md: mpx.vmhba1:C0:T4:L0, ssd: mpx.vmhba1:C0:T1:L0)
[...]

vsan.object_info
Prints informations about physical location and configuration from objects. The command output is very similar to the vsan.vm_object_info but it is used against a single object.

usage: object_info [opts] cluster obj_uuid...
Fetch information about a VSAN object
  cluster: Cluster on which to fetch the object info HostSystem or ClusterComputeResource
  obj_uuid:
           --skip-ext-attr, -s: Don't fetch extended attributes
  --include-detailed-usage, -i: Include detailed usage info
                    --help, -h: Show this message

Example 1 - Print physical location of a DOM Object:

/localhost/DC> vsan.object_info ~cluster 7e62c152-7dfb-c6e5-07b8-001b2193b9a4
2014-01-01 18:35:16 +0000: Fetching VSAN disk info from esx1.virten.lab (may take a moment) ...
2014-01-01 18:35:16 +0000: Fetching VSAN disk info from esx4.virten.lab (may take a moment) ...
2014-01-01 18:35:16 +0000: Fetching VSAN disk info from esx2.virten.lab (may take a moment) ...
2014-01-01 18:35:16 +0000: Fetching VSAN disk info from esx3.virten.lab (may take a moment) ...
2014-01-01 18:35:18 +0000: Done fetching VSAN disk infos
DOM Object: 7e62c152-7dfb-c6e5-07b8-001b2193b9a4 (owner: esx1.virten.lab, policy: hostFailuresToTolerate = 1, forceProvisioning = 1, proportionalCapacity = 100)
  Witness: c135c452-cd77-0733-1708-001b2193b9a4 (state: ACTIVE (5), host: esx4.virten.lab, md: t10.ATA_____WDC_WD300HLFSD2D03DFC1, ssd: t10.ATA_____SanDisk_SDSSDP064G)
  RAID_1
    Component: c135c452-2f04-0533-dbbc-001b2193b9a4 (state: ACTIVE (5), host: esx1.virten.lab, md: t10.ATA_____WDC_WD300HLFSD2D00NLR1, ssd: t10.ATA_____SanDisk_SDSSDP064G)
    Component: 7e62c152-763d-1400-2b06-001b2193b9a4 (state: ACTIVE (5), host: esx2.virten.lab, md: t10.ATA_____WDC_WD3000HLFS2D01G6U1, ssd: t10.ATA_____SanDisk_SDSSDP064G)

vsan.object_reconfigure
Changes the policy from DOM objects. To use this command, you need to know the object UUID which can be identified with vsan.cmmds_find or vsan.vm_object_info. Available policy options are:

  • hostFailuresToTolerate (Number of failures to tolerate)
  • forceProvisioning (If VSAN can't fulfill the policy requirements for an object, it will still deploy it)
  • stripeWidth (Number of disk stripes per object)
  • cacheReservation (Flash read cache reservation)
  • proportionalCapacity (Object space reservation)

Be careful to keep existing policies. Always specify all options. The policy has to be defined in the following format:
'(("hostFailuresToTolerate" i1) ("forceProvisioning" i1))'

usage: object_reconfigure [opts] cluster obj_uuid...
Reconfigure a VSAN object
  cluster: Cluster on which to execute the reconfig HostSystem or ClusterComputeResource
  obj_uuid: Object UUID
  --policy, -p <s>: New policy
        --help, -h: Show this message

Example 1 - Change the disk policy to tolerate 2 host failures.
Current policy is hostFailuresToTolerate = 1, stripeWidth = 1

/localhost/DC> vsan.object_reconfigure ~cluster 5078bd52-2977-8cf9-107c-00505687439c -p '(("hostFailuresToTolerate" i2) ("stripeWidth" i1))'

Example 2 - Disable Force provisioning. Current policy is hostFailuresToTolerate = 1, stripeWidth = 1

/localhost/DC> vsan.object_reconfigure ~cluster 5078bd52-2977-8cf9-107c-00505687439c -p '(("hostFailuresToTolerate" i1) ("stripeWidth" i1) ("forceProvisioning" i0))'

Example 3 - Locate and change VM disk policies

/localhost/DC> vsan.vm_object_info ~vm
VM perf1:
  Namespace directory
[...]
  Disk backing: [vsanDatastore] 6978bd52-4d92-05ed-dad2-005056871792/vma.virten.lab.vmdk
    DOM Object: 7e78bd52-7595-1716-85a2-005056871792 (owner: esx1.virtenlab, policy: hostFailuresToTolerate = 1, stripeWidth = 2, forceProvisioning = 1)
      Witness: aee5bd52-7443-177b-74a8-005056871792 (state: ACTIVE (5), host: esx2.virten.lab, md: mpx.vmhba1:C0:T4:L0, ssd: mpx.vmhba1:C0:T1:L0)
      RAID_1
        RAID_0
          Component: 36debd52-7390-a05d-9225-005056871792 (state: ACTIVE (5), esx3.virten.lab, md: mpx.vmhba1:C0:T2:L0, ssd: mpx.vmhba1:C0:T1:L0)
          Component: 36debd52-a9b8-965d-03a6-005056871792 (state: ACTIVE (5), esx3.virten.lab, md: mpx.vmhba1:C0:T4:L0, ssd: mpx.vmhba1:C0:T1:L0)
        RAID_0
          Component: 7f78bd52-2d59-c558-09f9-005056871792 (state: ACTIVE (5), esx1.virten.lab, md: mpx.vmhba1:C0:T2:L0, ssd: mpx.vmhba1:C0:T1:L0)
          Component: 7f78bd52-d827-c458-9d94-005056871792 (state: ACTIVE (5), esx1.virten.lab, md: mpx.vmhba1:C0:T4:L0, ssd: mpx.vmhba1:C0:T1:L0)

/localhost/DC> vsan.object_reconfigure ~cluster 7e78bd52-7595-1716-85a2-005056871792 -p '(("hostFailuresToTolerate" i1) ("stripeWidth" i1) ("forceProvisioning" i1))'
Reconfiguring '7e78bd52-7595-1716-85a2-005056871792' to (("hostFailuresToTolerate" i1) ("stripeWidth" i1) ("forceProvisioning" i1))

All reconfigs initiated. Synching operation may be happening in the background

vsan.vmdk_stats
Displays read cache and capacity stats for vmdks.

usage: vmdk_stats cluster_or_host vms...
Print read cache and capacity stats for vmdks.
Disk Capacity (GB):
Disk Size: Size of the vmdk
Used Capacity: MD capacity used by this vmdk
Data Size: Size of data on this vmdk
Read Cache (GB):
Used: RC used by this vmdk
Reserved: RC reserved by this vmdk
  cluster_or_host: Path to a ClusterComputeResource or HostSystem
  vms: Path to a VirtualMachine
  --help, -h:   Show this message

Troubleshooting

vsan.obj_status_report
Provides information about objects and their health status. With this command, you can identify that all object components are healthy, which means that witness and all mirrors are available and synced. It also identifies possibly orphaned objects.

usage: obj_status_report [opts] cluster_or_host
Print component status for objects in the cluster.
  cluster_or_host: Path to a ClusterComputeResource or HostSystem
           --print-table, -t: Print a table of object and their 
                              status, default all objects
      --filter-table, -f <s>: Filter the obj table based on status 
                              displayed in histogram, e.g. 2/3
           --print-uuids, -u: In the table, print object UUIDs 
                              instead of vmdk and vm paths
  --ignore-node-uuid, -i <s>: Estimate the status of objects if 
                              all comps on a given host were healthy.
                  --help, -h: Show this message

Example 1 - Simple component status histogram.
We can see 45 objects with 3/3 healthy components and 23 objects with 7/7 healthy components. With default policies, 3/3 are disks (2 mirror+witness) and 7/7 are namespace directories. We can also see an orphand object in this example.

/localhost/DC> vsan.obj_status_report ~cluster
2014-01-03 19:10:13 +0000: Querying all VMs on VSAN ...
2014-01-03 19:10:13 +0000: Querying all objects in the system from esx1.virten.lab ...
2014-01-03 19:10:14 +0000: Querying all disks in the system ...
2014-01-03 19:10:15 +0000: Querying all components in the system ...
2014-01-03 19:10:15 +0000: Got all the info, computing table ...

Histogram of component health for non-orphaned objects

+-------------------------------------+------------------------------+
| Num Healthy Comps / Total Num Comps | Num objects with such status |
+-------------------------------------+------------------------------+
| 3/3                                 |  45                          |
| 7/7                                 |  23                          |
+-------------------------------------+------------------------------+
Total non-orphans: 68

Histogram of component health for possibly orphaned objects

+-------------------------------------+------------------------------+
| Num Healthy Comps / Total Num Comps | Num objects with such status |
+-------------------------------------+------------------------------+
| 1/3                                 |  1                           |
+-------------------------------------+------------------------------+
Total orphans: 1

Example 2 - Add a table with all object and their status to the command output. That output reveals which object actually is orphaned.

/localhost/DC> vsan.obj_status_report ~cluster -t
2014-01-03 19:42:13 +0000: Querying all VMs on VSAN ...
2014-01-03 19:42:13 +0000: Querying all objects in the system from esx1.virten.lab ...
2014-01-03 19:42:14 +0000: Querying all disks in the system ...
2014-01-03 19:42:15 +0000: Querying all components in the system ...
2014-01-03 19:42:16 +0000: Got all the info, computing table ...

Histogram of component health for non-orphaned objects

+-------------------------------------+------------------------------+
| Num Healthy Comps / Total Num Comps | Num objects with such status |
+-------------------------------------+------------------------------+
| 3/3                                 |  45                          |
| 7/7                                 |  23                          |
+-------------------------------------+------------------------------+
Total non-orphans: 68

Histogram of component health for possibly orphaned objects

+-------------------------------------+------------------------------+
| Num Healthy Comps / Total Num Comps | Num objects with such status |
+-------------------------------------+------------------------------+
| 1/3                                 |  1                           |
+-------------------------------------+------------------------------+
Total orphans: 1

+-----------------------------------------------------------------------------+---------+---------------------------+
| VM/Object                                                                   | objects | num healthy / total comps |
+-----------------------------------------------------------------------------+---------+---------------------------+
| perf9                                                                       | 1       |                           |
|    [vsanDatastore] 735ec152-da7c-64b1-ebfe-eca86bf99b3f/perf9.vmx           |         | 7/7                       |
| perf8                                                                       | 1       |                           |
|    [vsanDatastore] 195ec152-92a3-491b-801d-eca86bf99b3f/perf8.vmx           |         | 7/7                       |
[...]
+-----------------------------------------------------------------------------+---------+---------------------------+
| Unassociated objects                                                        |         |                           |
[...]
|    795dc152-6faa-9ae7-efe5-001b2193b9a4                                     |         | 3/3                       |
|    d068ab52-b882-c6e8-32ca-eca86bf99b3f                                     |         | 1/3*                      |
|    ce5dc152-c5f6-3efb-9a44-001b2193b3b0                                     |         | 3/3                       |
+-----------------------------------------------------------------------------+---------+---------------------------+

+------------------------------------------------------------------+
| Legend: * = all unhealthy comps were deleted (disks present)     |
|         - = some unhealthy comps deleted, some not or can't tell |
|         no symbol = We cannot conclude any comps were deleted    |
+------------------------------------------------------------------+

Example 3 - Add a filtered table with unhealthy components only. We filter the table to show 1/3 health components only

/localhost/DC> vsan.obj_status_report ~cluster -t -f 1/3 
[...]
+-----------------------------------------+---------+---------------------------+
| VM/Object                               | objects | num healthy / total comps |
+-----------------------------------------+---------+---------------------------+
| Unassociated objects                    |         |                           |
|    d068ab52-b882-c6e8-32ca-eca86bf99b3f |         | 1/3*                      |
+-----------------------------------------+---------+---------------------------+
[...]
/localhost/DC> vsan.obj_status_report ~cluster -t -u 
[...]
+-----------------------------------------+---------+---------------------------+
| VM/Object                               | objects | num healthy / total comps |
+-----------------------------------------+---------+---------------------------+
| perf9                                   | 1       |                           |
|    735ec152-da7c-64b1-ebfe-eca86bf99b3f |         | 7/7                       |
| perf8                                   | 1       |                           |
|    195ec152-92a3-491b-801d-eca86bf99b3f |         | 7/7                       |
| perf11                                  | 1       |                           |
|    d65ec152-7d27-f441-39bb-eca86bf99b3f |         | 7/7                       |
[...]vsan.check_state
Checks state of VMs and VSAN objects. This command can also re-register vms where objects are out of sync. I can't reproduce to get VMs out of sync, so I couldn't test that for now. I am going to update that post when i have any further information. 

Example 1 - Check state
/localhost/DC>  vsan.check_state ~cluster
2014-01-03 19:53:36 +0000: Step 1: Check for inaccessible VSAN objects
Detected 1 objects to not be inaccessible
Detected d068ab52-b882-c6e8-32ca-eca86bf99b3f on esx2.virten.lab to be inaccessible

2014-01-03 19:53:37 +0000: Step 2: Check for invalid/inaccessible VMs

2014-01-03 19:53:37 +0000: Step 3: Check for VMs for which VC/hostd/vmx are out of sync
Did not find VMs for which VC/hostd/vmx are out of sync

vsan.fix_renamed_vms
This command fixes virtual machines that are renamed by the vCenter in case of storage inaccessibility when they are get renamed to their vmx file path.
It is a best effort command, as the real VM name is unknown.

usage: fix_renamed_vms vms...
This command can be used to rename some VMs which get renamed by the VC in case of storage inaccessibility. It is
possible for some VMs to get renamed to vmx file path. eg. "/vmfs/volumes/vsanDatastore/foo/foo.vmx". This command will
rename this VM to "foo". This is the best we can do. This VM may have been named something else but we have no way to
know. In this best effort command, we simply rename it to the name of its config file (without the full path and .vmx
extension ofcourse!).
  vms: Path to a VirtualMachine
  --help, -h: Show this message

Example 1 - Fix a renamed VM
vsan-rename-vm

/localhost/DC>  vsan.fix_renamed_vms ~/vms/%2fvmfs%2fvolumes%2fvsanDatastore%2fvma.virten.lab%2fvma.virten.lab.vmx/
Continuing this command will rename the following VMs:
%2fvmfs%2fvolumes%2fvsanDatastore%2fvma.virten.lab%2fvma.virten.lab.vmx -> vma.virten.lab
Do you want to continue [y/N]?
y
Renaming...
Rename %2fvmfs%2fvolumes%2fvsanDatastore%2fvma.virten.lab%2fvma.virten.lab.vmx: success

vsan.reapply_vsan_vmknic_config
Re-enables VSAN on vmk ports. Could be useful when you have network configuration problems in your VSAN Cluster.

usage: reapply_vsan_vmknic_config [opts] host...
Unbinds and rebinds VSAN to its vmknics
  host: Path to a HostSystem
  --vmknic, -v <s>: Refresh a specific vmknic. default is all vmknics
     --dry-run, -d: Do a dry run: Show what changes would be made
        --help, -h: Show this message

Example 1 - Run the command in dry mode

/localhost/DC> vsan.reapply_vsan_vmknic_config -d ~esx
Host: esx1.virten.lab
  Would reapply config of vmknic vmk1:
    AgentGroupMulticastAddress: 224.2.3.4
    AgentGroupMulticastPort: 23451
    IPProtocol: IPv4
    InterfaceUUID: 776ca852-6660-c6d8-c9f4-001b2193b9a4
    MasterGroupMulticastAddress: 224.1.2.3
    MasterGroupMulticastPort: 12345
    MulticastTTL: 5

Example 2 - Unbinds and rebinds VSAN on a host

/localhost/DC> vsan.reapply_vsan_vmknic_config ~esx
Host: esx1.virten.lab
  Reapplying config of vmk1:
    AgentGroupMulticastAddress: 224.2.3.4
    AgentGroupMulticastPort: 23451
    IPProtocol: IPv4
    InterfaceUUID: 776ca852-6660-c6d8-c9f4-001b2193b9a4
    MasterGroupMulticastAddress: 224.1.2.3
    MasterGroupMulticastPort: 12345
    MulticastTTL: 5
  Unbinding VSAN from vmknic vmk1 ...
  Rebinding VSAN to vmknic vmk1 ...

vsan.vm_perf_stats
Displays performance statistics from a virtual machine. The following metrics are supported:

  • IOPS (read/write)
  • Throughput‎ in KB/s (read/write
  • Latency in ms (read/write)
usage: vm_perf_stats [opts] vms...
VM perf stats
  vms: Path to a VirtualMachine
  --interval, -i <i>: Time interval to compute average over (default: 20)
  --show-objects, -s: Show objects that are part of VM
          --help, -h: Show this message

Example 1 - Display performance stats (Default: 20sec interval)

/localhost/DC> san.vm_perf_stats ~vm
2014-01-03 20:23:44 +0000: Querying info about VMs ...
2014-01-03 20:23:44 +0000: Querying VSAN objects used by the VMs ...
2014-01-03 20:23:45 +0000: Fetching stats counters once ...
2014-01-03 20:23:46 +0000: Sleeping for 20 seconds ...
2014-01-03 20:24:06 +0000: Fetching stats counters again to compute averages ...
2014-01-03 20:24:07 +0000: Got all data, computing table
+-----------+--------------+------------------+--------------+
| VM/Object | IOPS         | Tput (KB/s)      | Latency (ms) |
+-----------+--------------+------------------+--------------+
| win7      | 325.1r/80.6w | 20744.8r/5016.8w | 2.4r/12.2w   |
+-----------+--------------+------------------+--------------+

Example 2 - Display all VM objects performance stats with an interval of 5.

/localhost/DC> vsan.vm_perf_stats ~vm --show-objects --interval=5
2014-01-03 20:29:08 +0000: Querying info about VMs ...
2014-01-03 20:29:08 +0000: Querying VSAN objects used by the VMs ...
2014-01-03 20:29:09 +0000: Fetching stats counters once ...
2014-01-03 20:29:09 +0000: Sleeping for 5 seconds ...
2014-01-03 20:29:14 +0000: Fetching stats counters again to compute averages ...
2014-01-03 20:29:15 +0000: Got all data, computing table
+-----------------------------------------------------+-------------+----------------+--------------+
| VM/Object                                           | IOPS        | Tput (KB/s)    | Latency (ms) |
+-----------------------------------------------------+-------------+----------------+--------------+
| win7                                                |             |                |              |
|    ddc4a952-c91c-a777-771f-001b2193b9a4/win7.vmx    | 0.0r/0.3w   | 0.0r/0.2w      | 0.0r/26.1w   |
|    ddc4a952-c91c-a777-771f-001b2193b9a4/win7.vmdk   | 166.6r/3.9w | 10597.2r/13.2w | 5.2r/23.5w   |
|    ddc4a952-c91c-a777-771f-001b2193b9a4/win7_1.vmdk | 0.0r/72.6w  | 0.0r/4583.7w   | 0.0r/13.1w   |
+-----------------------------------------------------+-------------+----------------+--------------+

vsan.clear_disks_cache
Clears cached disk informations. $disksCache is an internal variable. Usually, this command is not required. If you encounter problems it might help.

usage: clear_disks_cache
Clear cached disks information
  --help, -h: Show this message

Example 1 - Display performance stats. (No outpu)

/localhost/DC> vsan.clear_disks_cache

vsan.lldpnetmap
Displays the mapping between physical network interface and physical switchs by using using Link Layer Discovery Protocol (LLDP).

usage: lldpnetmap hosts_and_clusters...
Gather LLDP mapping information from a set of hosts
  hosts_and_clusters: Path to a HostSystem or ClusterComputeResource
  --help, -h: Show this message

Example 1 - Display LLDP information from all hosts

/localhost/DC> vsan.lldpnetmap ~cluster
2014-12-30 18:55:40 +0000: This operation will take 30-60 seconds ...
+-------------------+--------------+
| Host              | LLDP info    |
+-------------------+--------------+
| esx2.virten.local | s320: vmnic1 |
| esx3.virten.local | s320: vmnic1 |
| esx1.virten.local | s320: vmnic1 |
+-------------------+--------------+

vsan.recover_spbm
This command recovers Storage Policy Based Management associations.

usage: recover_spbm [opts] cluster_or_host
SPBM Recovery
  cluster_or_host: Path to a ClusterComputeResource or HostSystem
  --dry-run, -d: Don't take any automated actions
    --force, -f: Answer all question with 'yes'
     --help, -h: Show this message

Example 1 - Recover SPBM profile association

/localhost/DC> vsan.recover_spbm -f ~cluster
2014-12-30 19:02:53 +0000: Fetching Host info
2014-12-30 19:02:53 +0000: Fetching Datastore info
2014-12-30 19:02:53 +0000: Fetching VM properties
2014-12-30 19:02:53 +0000: Fetching policies used on VSAN from CMMDS
2014-12-30 19:02:53 +0000: Fetching SPBM profiles
2014-12-30 19:02:54 +0000: Fetching VM <-> SPBM profile association
2014-12-30 19:02:54 +0000: Computing which VMs do not have a SPBM Profile ...
2014-12-30 19:02:54 +0000: Fetching additional info about some VMs
2014-12-30 19:02:54 +0000: Got all info, computing after 0.66 sec
2014-12-30 19:02:54 +0000: Done computing
SPBM Profiles used by VSAN:
+---------------------+---------------------------+
| SPBM ID             | policy                    |
+---------------------+---------------------------+
| Not managed by SPBM | proportionalCapacity: 100 |
|                     | hostFailuresToTolerate: 1 |
+---------------------+---------------------------+
| Not managed by SPBM | hostFailuresToTolerate: 1 |
+---------------------+---------------------------+

vsan.support_information
Use this command to gather information for support cases at VMware. This commands runs several other commands:

  • vsan.cluster_info VSAN-Cluster
  • vsan.host_info
  • vsan.vm_object_info
  • vsan.vm_object_info
  • vsan.disks_info
  • vsan.disks_stats
  • vsan.check_limits VSAN-Cluster
  • vsan.check_state VSAN-Cluster
  • vsan.check_state VSAN-Cluster
  • vsan.lldpnetmap VSAN-Cluster
  • vsan.obj_status_report VSAN-Cluster
  • vsan.resync_dashboard VSAN-Cluster
  • vsan.disk_object_info VSAN-Cluster, disk_uuids
usage: support_information dc_or_clust_conn
Command to collect vsan support information
  dc_or_clust_conn: Path to a RbVmomi::VIM or Datacenter or ClusterComputeResource
  --help, -h: Show this message

Example 1 - Gather Support Information

/localhost/DC> vsan.support_information ~
VMware virtual center DAE11A48-50D6-4D6F-91F5-1E1D349A6917
*** command> vsan.support_information DC
************* BEGIN Support info for datacenter DC *************
************* BEGIN Support info for cluster VSAN-Cluster *************

*** command>vsan.cluster_info VSAN-Cluster
*** command>vsan.host_info 
*** command>vsan.vm_object_info
*** command>vsan.vm_object_info
*** command>vsan.disks_info
*** command>vsan.disks_stats
*** command>vsan.check_limits VSAN-Cluster
*** command>vsan.check_state VSAN-Cluster
*** command>vsan.check_state VSAN-Cluster
*** command>vsan.lldpnetmap VSAN-Cluster
*** command>vsan.obj_status_report VSAN-Cluster
*** command>vsan.resync_dashboard VSAN-Cluster
*** command>vsan.disk_object_info VSAN-Cluster, disk_uuids

************* END Support info for cluster VSAN-Cluster *************

************* END Support info for datacenter DC *************

VSAN Observer

Basics All commands can be configured with an Interval in which Observer will collect stats and a maximum runtime. Default is to collect stats each 60 seconds and run 2 hours. Keep in mind that VSAN Observer runs from memory when adjusting runtime settings. Options to change the default settings for runtime and interval are -i and -m.

usage: observer [opts] cluster_or_host
Run observer
  cluster_or_host: Path to a ClusterComputeResource or HostSystem
                --filename, -f <s>: Output file path
                    --port, -p <i>: Port on which to run webserver (default: 8010)
               --run-webserver, -r: Run a webserver to view live stats
                       --force, -o: Apply force
  --keep-observation-in-memory, -k: Keep observed stats in memory even when 
                                    commands ends. Allows to resume later
    --generate-html-bundle, -g <s>: Generates an HTML bundle after completion. 
                                    Pass a location
                --interval, -i <i>: Interval (in sec) in which to collect 
                                    stats (default: 60)
             --max-runtime, -m <i>: Maximum number of hours to collect stats. 
                                    Caps memory usage. (Default: 2)
                 --forever, -e <s>: Runs until stopped. Every --max-runtime 
                                    intervals retires snapshot to disk. Pass 
                                    a location
                    --no-https, -n: Don't use HTTPS and don't require login. 
                                    Warning: Insecure
        --max-diskspace-gb, -a <i>: Maximum disk space (in GB) to use in forever 
                                    mode. Deletes old data periodically (default: 5)
                        --help, -h: Show this message


Dump Data in JSON
VSAN Observer queries stats from each ESXi host in the JSON format. That output can be written to a file to be used by other tools. In each collection interval, stats are gathered from all hosts and written to a new line.

Example: I am going to run a single collection, and stop them with CTRL+C after the first time:

/localhost/DC> vsan.observer ~cluster --filename /tmp/observer.json
Press + to stop observing at any point ...
2014-01-09 20:15:56 +0000: Collect one inventory snapshot
Query VM properties: 0.16 sec
Query Stats on esx3.virten.lab: 0.65 sec (on ESX: 0.16, json size: 33KB)
Query Stats on esx2.virten.lab: 2.05 sec (on ESX: 0.51, json size: 461KB)
Query Stats on esx4.virten.lab: 2.19 sec (on ESX: 1.25, json size: 409KB)
Query Stats on esx1.virten.lab: 2.13 sec (on ESX: 0.50, json size: 472KB)
Query CMMDS from esx1.virten.lab: 1.84 sec (json size: 349KB)
2014-01-09 20:16:18 +0000: Collection took 21.98s, sleeping for 38.02s
2014-01-09 20:16:18 +0000: Press + to stop observing
^C2014-01-09 20:16:44 +0000: Execution interrupted, wrapping up ...

To format json output, i use python -mjson.tools:

vc:~ # cat /tmp/observer.json | python -mjson.tool |less

The output looks like this, but be warned, there are lots of data. One collection of my small 4 Node Cluster generates 110,000 lines:

[...]
       "vsi": {
            "esx1.virten.lab": {
                "cbrc": {
                    "dcacheStats": null
                },
                "cbrc-taken": 1389298607.2460959,
                "disks-taken": 1389298607.2457359,
                "disks.stats": {
                    "SanDisk_SDSSDP064G": {
                        "atsOps": 0,
                        "blocksCloneRead": 0,
                        "blocksCloneWrite": 0,
                        "blocksDeleted": 0,
                        "blocksRead": 36723527,
                        "blocksWritten": 50546140,
                        "blocksZeroed": 0,
                        "cloneReadOps": 0,
                        "cloneWriteOps": 0,
                        "commands": 3558146,
                        "dAvgLatency": 3748,
                        "dAvgMoving": 1288,
                        "deleteOps": 0,
                        "failedAtsOps": 0,
[...]

Generate HTML bundle
To visualize the huge amount of data, VSAN Observer can generate a HTML bundle after completion:

/localhost/DC> vsan.observer ~cluster --generate-html-bundle /tmp/
Press + to stop observing at any point ...

2014-01-09 20:37:44 +0000: Collect one inventory snapshot
Query VM properties: 0.17 sec
Query Stats on esx3.virten.lab: 0.50 sec (on ESX: 0.16, json size: 33KB)
Query Stats on esx2.virten.lab: 2.02 sec (on ESX: 0.51, json size: 461KB)
Query Stats on esx1.virten.lab: 2.15 sec (on ESX: 0.54, json size: 472KB)
Query Stats on esx4.virten.lab: 2.99 sec (on ESX: 2.28, json size: 409KB)
Query CMMDS from esx1.virten.lab: 1.84 sec (json size: 349KB)
2014-01-09 20:38:06 +0000: Live-Processing inventory snapshot
2014-01-09 20:38:06 +0000: Collection took 21.93s, sleeping for 38.07s
2014-01-09 20:38:06 +0000: Press + to stop observing
^C2014-01-09 20:38:12 +0000: Execution interrupted, wrapping up ...
2014-01-09 20:38:12 +0000: Writing out an HTML bundle to /tmp/vsan-observer-2014-01-09.20-38-12.tar.gz ...
2014-01-09 20:38:12 +0000: Writing statsdump for system mem ...
2014-01-09 20:38:12 +0000: Writing statsdump for pnics ...
2014-01-09 20:38:12 +0000: Writing statsdump for slabs ...
2014-01-09 20:38:12 +0000: Writing statsdump for heaps ...
2014-01-09 20:38:12 +0000: Writing statsdump for pcpus ...
2014-01-09 20:38:12 +0000: Writing statsdump for ssds ...
2014-01-09 20:38:12 +0000: Writing statsdump for worldlets ...
2014-01-09 20:38:12 +0000: Writing statsdump for helper worlds ...
2014-01-09 20:38:12 +0000: Writing statsdump for DOM ...
2014-01-09 20:38:12 +0000: Writing statsdump for LSOM components ...
2014-01-09 20:38:12 +0000: Writing statsdump for LSOM hosts ...
2014-01-09 20:38:12 +0000: Writing statsdump for PLOG disks ...
2014-01-09 20:38:12 +0000: Writing statsdump for LSOM disks ...
2014-01-09 20:38:12 +0000: Writing statsdump for CBRC ...
2014-01-09 20:38:12 +0000: Writing statsdump for VMs ...
2014-01-09 20:38:12 +0000: Writing statsdump for VSCSI ...
2014-01-09 20:38:12 +0000: Writing statsdump for Physical disks ...

VSAN Observer Live View
VSAN Observer also has an implemented web server that allows to view the same output as html-bundle live. Just start it and open http://<vcenter>:8010 in your browser:

/localhost/DC> vsan.observer ~cluster --run-webserver --force 
[2014-01-09 20:44:08] INFO  WEBrick 1.3.1
[2014-01-09 20:44:08] INFO  ruby 1.9.2 (2011-07-09) [x86_64-linux]
[2014-01-09 20:44:08] WARN  TCPServer Error: Address already in use - bind(2)
Press + to stop observing at any point ...

2014-01-09 20:44:08 +0000: Collect one inventory snapshot
[2014-01-09 20:44:08] INFO  WEBrick::HTTPServer#start: pid=26029 port=8010

The Webinterface comes up with 8 Views. I am going to cover the most useful stats for now:vsan-observer-menu

  • SAN Client
  • VSAN Disks
  • VSAN Disks (deep-dive)
  • PCPU
  • Memory
  • Distribution
  • DOM Owner
  • VMs

vsan-client

SAN Client
This view shows VSAN statistics from the view point of the host. The view represents the performance as seen by VMs running on the hosts for which statistics are shown. You should see balanced statistics in that view. To balance the workload you can try to use vMotion to move VMs to other ESXi hosts in the Cluster. This might help, but does not always solve the problem because the performance issue can be caused by any disk on any host in the VSAN cluster.

VSAN Disks
This view shows aggregated statistics from all disks in an host. This might help to identify the ESXi host which causes contention. 

VSAN Disks (deep-dive)
This view shows detailed information about physical disks on each host. This allows to further identify when single disks are causing contention. You can view latency and IOPS from all disks (SSD and MD) and special metrics like RC Hit rate and WriteBuffer load.

VM
This view allows to analyse single VMs. You can drill down each virtual disk and see where it is physically located. You can see objects and components performance metrics.

  1. Manage Virtual SAN with RVC – Complete Guide | Storage CH Blog - pingback on January 5, 2015 at 7:52 am
  2. De VSAN 6.1 à VSAN 6.2 … quelle aventure ! – vBlog.io - pingback on March 18, 2016 at 8:54 pm
  3. I noticed after removing a disk group from a host and re-adding it back, it now has N/A for the DisplayName and Host column; when running vsan.disks_stats I was instructed by VMware support to do this, due to invalid components. It did clean up the error, but I now have N/A for Displayname and Host. VSAN seems to be in good standing. Is there a way to add this information back for the vsan.disks_stats cmd? Thanks.

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Trackbacks and Pingbacks: