Skip to content

Cannot remove datastore * because file system is busy.

The following error message appears when you try to delete or unmount a VMFS datastore:

The resource Datastore Name: * VMFS uuid: * is in use.

Cannot remove datastore 'Datastore Name: * VMFS uuid: *' because file system is busy. Correct the problem and retry the operation.

Cannot-remove-datastore- because-file-system-is-busy

ESXi 5.5 has a new feature to store coredumps in a file residing on a datastore. It may sometimes create this file automatically and thus blocking datastores from being deleted. It also creates a vsantraces directory which blocks a datastore.

Dumpfile Removal
Check for dump files. You can run this command from any ESXi host with access to the datastore:

~ # esxcli system coredump file list
 Path                                                  Active Configured       Size
 ----------------------------------------------------- ------ ---------- ----------
 /vmfs/volumes/Datastore/vmkdump/684938663845.dumpfile  false      false 1714421760
 /vmfs/volumes/Datastore/vmkdump/684938663233.dumpfile  false      false 1714421760
 /vmfs/volumes/Datastore/vmkdump/684938663533.dumpfile  false      false 1714421760

The output shows that I have 3 dump files which are blocking my datastore. Only the owning ESXi host can disable and delete them, so you have to find out which ESXi is responsible for each file:

~ # vmkfstools -D /vmfs/volumes/Datastore/vmkdump/684938663845.dumpfile
 Lock [type 10c00001 offset 200392704 v 10, hb offset 3875328
 gen 3, mode 1, owner 52ebd042-43b191f0-0173-005056871792 mtime 250
 num 0 gblnum 0 gblgen 0 gblbrk 0]
 Addr <4, 447, 0>, gen 1, links 1, type reg, flags 0, uid 0, gid 0, mode 600
 len 1714421760, nb 1635 tbz 0, cow 0, newSinceEpoch 1635, zla 3, bs 1048576

You can see the UUID from the ESXi host that has locked the file. (Side note: The last part of the UUID is determined by the MAC address of vmnic0. 005056871792 = 00:50:56:87:17:92). To quickly identify the host, you can use the following PowerCLI Script to list all ESXi hosts, with their UUID:

Get-View -ViewType HostSystem -Propert Name, hardware.systeminfo | select { $_.name, $_.hardware.systeminfo.uuid }

If you cannot find it, try to locate the MAC address:

Get-VMHostNetworkAdapter |? {$_.Mac -eq "xx:xx:xx:xx:xx:xx"} | select VMhost, Name, Mac

To remove the coredump file, connect to the ESXi host with SSH and use the esxli system coredump file remove command. This will remove the configured active coredump file:

~ # esxcli system coredump file remove --force

If you have many hosts, you can also use this little PowerCLI script to quickly remove all coredump files. You have to be connected to the vCenter:

Get-VMHost | % {
 $esxcli = get-esxcli -vmhost $_
 $esxcli.system.coredump.file.remove($null, $true)
}

Check this post if you want to remove dumpfiles permanently.

vsantrace Removal
vsantraced is also a potential datastore blocker. You can verify this by listing open files:

~ # lsof |grep vsantraced |grep volumes
34412 vsantraced FILE 3 /vmfs/volumes/c44fd830190/vsantraces/vsantraces--2014-02-28T12h36m36s899.gz

To remove the log, simply stop vsantraced, unmount the datastore and start it again:

~ # /etc/init.d/vsantraced stop

# Remove/Unmount Datastore

~ # /etc/init.d/vsantraced start

If you do not use Virtual SAN you can also disable vsantraced permanently:

# chkconfig vsantraced off

Scratch Location

Verify the value from the advanced setting ScratchConfig.CurrentScratchLocation  (ESXi > Configure > System > Advanced System Settings). If the ESxi host is used as Scratch Location, edit the value ScratchConfig.ConfiguredScratchLocation and reboot the host.

LUN removal checklist

  • No virtual machine, template, snapshot or CD/DVD image resides on the datastore
  • The datastore is not part of a Datastore Cluster
  • Storage I/O Control is disabled for the datastore
  • The datastore is not used for vSphere HA heartbeat
  • The LUN is not used as an RDM
  • The Datastore is not used as a scratch location
  • The Datastore is not used as VMkernel Dump file location (/vmkdump/)
  • The Datastore is not used as active vsantraced location (/vsantrace/)
  • The Datastore is not used as Scratch location
  • The Datastore is not used to store VM swap files.

VMware KB2004605 also assists in removing datastores in ESXi 5.x and 6.x

15 thoughts on “Cannot remove datastore * because file system is busy.”

  1. Do existence of Dump files prevent deleting a VMFS datastore? I don't see any technical reason why it does? Can you explain more?

    1. Yes, when the dump file is active ("esxcli system coredump file get" or "esxcfg-dumppart -l"). The file is actually not the dump itself, it is the location where a dump is placed when the ESXi host crashes. Since ESXi5.5 the coredump partition is limited to 100MB, so it may create this file.

        1. Not sure if it is "random", but Yes. When there is no local partition, it picks "any" datastore and puts the coredump file there.

  2. Pingback: Permanently disable ESXi 5.5 coredump file | Virten.net

  3. Thanks, worked for me too !!! :-)

    i also find out that issuing "umount" on the data store, reveals the ESX using it easier and faster then looking for mac addresses :-)

  4. Cheers for the tips - after installing 5.5 and migrating vm from local to SAN, all that was left blocking datastore removal were the dump files and vsantraces.

  5. even if swap files of virtual machine is present in datastore it wont allow you to remove datastore we need to move swap file to local store or any other location

Leave a Reply to yogesh Cancel reply

Your email address will not be published. Required fields are marked *