While installing ESXi updates, I noticed that on one of my hosts, the installation or removal of VIB packages fails with the following error message
# esxcli software vib install -d [package]
# esxcli software vib remove -n [package]
[InstallationError]
Failed to query file system stats: Errors:
Error getting data for filesystem on '/vmfs/volumes/59a83d9c-628c6ae0-7b35-f44d306ec05a': Cannot open volume: /vmfs/volumes/59a83d9c-628c6ae0-7b35-f44d306ec05a, skipping.
cause = Errors:
Error getting data for filesystem on '/vmfs/volumes/59a83d9c-628c6ae0-7b35-f44d306ec05a': Cannot open volume: /vmfs/volumes/59a83d9c-628c6ae0-7b35-f44d306ec05a, skipping.
Please refer to the log file for more details.
The device 59a83d9c-628c6ae0-7b35-f44d306ec05a was a non existing volume, referenced by a vffs mount. VFFS (Virtual Flash File System) was used in earlier ESXi releases by vSphere Flash Read Cache. I'm not sure where that comes from but this is how you can remove the stale mount:
First of all, use ls /vmfs/volumes/ to figure out what's wrong. The ID from the error message could be anything:
You can see that 59a83d9c-628c6ae0-7b35-f44d306ec05a is referenced by vffs-52329a0c-4660-19ce-c61f-7fbf9c717686, which is related to vFRC (vSphere Flash Read Cache).
Use esxcli storage vflash device list you can identify if any volumes are in or have been used with vFRC:
[root@esx3:/vmfs/volumes] esxcli storage vflash device list Name Size Is Local Is Used in vflash Eligibility -------------------------------------------------------------------------------- ------ -------- ----------------- ----------- naa.6589cfc000000ed999506ae18cee259a 524288 false false Is not local SSD device t10.NVMe____INTEL_MEMPEK1W016GA_____________________PHBT714605SA016D____00000001 13736 true true It has been configured for vflash t10.ATA_____Samsung_SSD_850_EVO_500GB_______________S2RANX0H547670E_____ 476940 true false Yes, this is a blank disk. naa.6589cfc000000656da382d1be31512d2 524288 false false Is not local SSD device
Apparently, the t10.NVMe____INTEL_MEMPEK1W016GA device was used with vFRC and still has configuration fragments. The drive is definitely not in use for caching, vsan, datastores or anything else. Let's check if it has partitions:
# ls -l /dev/disks |grep t10.NVMe____INTEL_MEMPEK1W016GA -rw------- 1 root root 14403239936 Nov 27 16:09 t10.NVMe____INTEL_MEMPEK1W016GA_____________________PHBT714605SA016D____00000001 -rw------- 1 root root 14402174464 Nov 27 16:09 t10.NVMe____INTEL_MEMPEK1W016GA_____________________PHBT714605SA016D____00000001:1
There is a partition which I don't need. This appears to be the old vFRC partition. Delete it:
# partedUtil delete /dev/disks/t10.NVMe____INTEL_MEMPEK1W016GA_____________________PHBT714605SA016D____00000001 1 # ls -l /dev/disks |grep t10.NVMe____INTEL_MEMPEK1W016GA -rw------- 1 root root 14403239936 Nov 27 16:10 t10.NVMe____INTEL_MEMPEK1W016GA_____________________PHBT714605SA016D____00000001
And it's gone. Now reboot the ESXi host to remove the stale mount.
# reboot
After ESXi is back up, verify that the volume is gone. You should now be able to install/remove software packages.
