Installation or Removal of VIB Packages in ESXi 7.0 fails with Error: Failed to query file system stats:

While installing ESXi updates, I noticed that on one of my hosts, the installation or removal of VIB packages fails with the following error message

# esxcli software vib install -d [package]
# esxcli software vib remove -n [package]
[InstallationError]
Failed to query file system stats: Errors:
Error getting data for filesystem on '/vmfs/volumes/59a83d9c-628c6ae0-7b35-f44d306ec05a': Cannot open volume: /vmfs/volumes/59a83d9c-628c6ae0-7b35-f44d306ec05a, skipping.
cause = Errors:
Error getting data for filesystem on '/vmfs/volumes/59a83d9c-628c6ae0-7b35-f44d306ec05a': Cannot open volume: /vmfs/volumes/59a83d9c-628c6ae0-7b35-f44d306ec05a, skipping.
Please refer to the log file for more details.

The device 59a83d9c-628c6ae0-7b35-f44d306ec05a was a non existing volume, referenced by a vffs mount. VFFS (Virtual Flash File System) was used in earlier ESXi releases by vSphere Flash Read Cache. I'm not sure where that comes from but this is how you can remove the stale mount:

First of all, use ls /vmfs/volumes/ to figure out what's wrong. The ID from the error message could be anything:

You can see that 59a83d9c-628c6ae0-7b35-f44d306ec05a is referenced by vffs-52329a0c-4660-19ce-c61f-7fbf9c717686, which is related to vFRC (vSphere Flash Read Cache).

Use esxcli storage vflash device list you can identify if any volumes are in or have been used with vFRC:

[root@esx3:/vmfs/volumes] esxcli storage vflash device list
Name                                                                                Size  Is Local  Is Used in vflash  Eligibility
--------------------------------------------------------------------------------  ------  --------  -----------------  -----------
naa.6589cfc000000ed999506ae18cee259a                                              524288     false              false  Is not local SSD device
t10.NVMe____INTEL_MEMPEK1W016GA_____________________PHBT714605SA016D____00000001   13736      true               true  It has been configured for vflash
t10.ATA_____Samsung_SSD_850_EVO_500GB_______________S2RANX0H547670E_____          476940      true              false  Yes, this is a blank disk.
naa.6589cfc000000656da382d1be31512d2                                              524288     false              false  Is not local SSD device

Apparently, the t10.NVMe____INTEL_MEMPEK1W016GA device was used with vFRC and still has configuration fragments. The drive is definitely not in use for caching, vsan, datastores or anything else. Let's check if it has partitions:

# ls -l /dev/disks |grep t10.NVMe____INTEL_MEMPEK1W016GA
-rw------- 1 root root 14403239936 Nov 27 16:09 t10.NVMe____INTEL_MEMPEK1W016GA_____________________PHBT714605SA016D____00000001
-rw------- 1 root root 14402174464 Nov 27 16:09 t10.NVMe____INTEL_MEMPEK1W016GA_____________________PHBT714605SA016D____00000001:1

There is a partition which I don't need. This appears to be the old vFRC partition. Delete it:

# partedUtil delete /dev/disks/t10.NVMe____INTEL_MEMPEK1W016GA_____________________PHBT714605SA016D____00000001 1
# ls -l /dev/disks |grep t10.NVMe____INTEL_MEMPEK1W016GA
-rw-------    1 root     root     14403239936 Nov 27 16:10 t10.NVMe____INTEL_MEMPEK1W016GA_____________________PHBT714605SA016D____00000001

And it's gone. Now reboot the ESXi host to remove the stale mount.

# reboot

After ESXi is back up, verify that the volume is gone. You should now be able to install/remove software packages.

 

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.