Cannot remove datastore * because file system is busy.

The following error message appears when you try to delete or unmount a VMFS datastore:

The resource Datastore Name: * VMFS uuid: * is in use.

Cannot remove datastore 'Datastore Name: * VMFS uuid: *' because file system is busy. Correct the problem and retry the operation.

Cannot-remove-datastore- because-file-system-is-busy

ESXi 5.5 has a new feature to store coredumps in a file residing on a datastore. It may sometimes create this file automatically and thus blocking datastores from being deleted. It also creates a vsantraces directory which blocks a datastore.

Dumpfile Removal
Check for dump files. You can run this command from any ESXi host with access to the datastore:

~ # esxcli system coredump file list
 Path                                                  Active Configured       Size
 ----------------------------------------------------- ------ ---------- ----------
 /vmfs/volumes/Datastore/vmkdump/684938663845.dumpfile  false      false 1714421760
 /vmfs/volumes/Datastore/vmkdump/684938663233.dumpfile  false      false 1714421760
 /vmfs/volumes/Datastore/vmkdump/684938663533.dumpfile  false      false 1714421760

The output shows that I have 3 dump files which are blocking my datastore. Only the owning ESXi host can disable and delete them, so you have to find out which ESXi is responsible for each file:

~ # vmkfstools -D /vmfs/volumes/Datastore/vmkdump/684938663845.dumpfile
 Lock [type 10c00001 offset 200392704 v 10, hb offset 3875328
 gen 3, mode 1, owner 52ebd042-43b191f0-0173-005056871792 mtime 250
 num 0 gblnum 0 gblgen 0 gblbrk 0]
 Addr <4, 447, 0>, gen 1, links 1, type reg, flags 0, uid 0, gid 0, mode 600
 len 1714421760, nb 1635 tbz 0, cow 0, newSinceEpoch 1635, zla 3, bs 1048576

You can see the UUID from the ESXi host that has locked the file. (Side note: The last part of the UUID is determined by the MAC address of vmnic0. 005056871792 = 00:50:56:87:17:92). To quickly identify the host, you can use the following PowerCLI Script to list all ESXi hosts, with their UUID:

Get-View -ViewType HostSystem -Propert Name, hardware.systeminfo | select { $_.name, $_.hardware.systeminfo.uuid }

If you cannot find it, try to locate the MAC address:

Get-VMHostNetworkAdapter |? {$_.Mac -eq "xx:xx:xx:xx:xx:xx"} | select VMhost, Name, Mac

To remove the coredump file, connect to the ESXi host with SSH and use the esxli system coredump file remove command. This will remove the configured active coredump file:

~ # esxcli system coredump file remove --force

If you have many hosts, you can also use this little PowerCLI script to quickly remove all coredump files. You have to be connected to the vCenter:

Get-VMHost | % {
 $esxcli = get-esxcli -vmhost $_
 $esxcli.system.coredump.file.remove($null, $true)
}

Check this post if you want to remove dumpfiles permanently.

vsantrace Removal
vsantraced is also a potential datastore blocker. You can verify this by listing open files:

~ # lsof |grep vsantraced |grep volumes
34412 vsantraced FILE 3 /vmfs/volumes/c44fd830190/vsantraces/vsantraces--2014-02-28T12h36m36s899.gz

To remove the log, simply stop vsantraced, unmount the datastore and start it again:

~ # /etc/init.d/vsantraced stop

# Remove/Unmount Datastore

~ # /etc/init.d/vsantraced start

If you do not use Virtual SAN you can also disable vsantraced permanently:

# chkconfig vsantraced off

 

LUN removal checklist

  • No virtual machine, template, snapshot or CD/DVD image resides on the datastore
  • The datastore is not part of a Datastore Cluster
  • Storage I/O Control is disabled for the datastore
  • The datastore is not used for vSphere HA heartbeat
  • The LUN is not used as a RDM
  • The Datastore is not used as a scratch location
  • The Datastore is not used as VMkernel Dump file location (/vmkdump/)
  • The Datastore is not used as active vsantraced location (/vsantrace/)

VMware KB2004605 also assists on removing datastores in ESXi 5.x

Leave a comment ?

14 Comments.

  1. Do existence of Dump files prevent deleting a VMFS datastore? I don't see any technical reason why it does? Can you explain more?

    • Yes, when the dump file is active ("esxcli system coredump file get" or "esxcfg-dumppart -l"). The file is actually not the dump itself, it is the location where a dump is placed when the ESXi host crashes. Since ESXi5.5 the coredump partition is limited to 100MB, so it may create this file.

      • So it just picks a random datastore to store the coredump when a host crashes?

        • Not sure if it is "random", but Yes. When there is no local partition, it picks "any" datastore and puts the coredump file there.

  2. Permanently disable ESXi 5.5 coredump file | Virten.net - pingback on February 27, 2014 at 9:09 pm
  3. Thanks, It's worked for me !!! Nice guide ! :grin:

  4. Thanks, worked for me too !!! :-)

    i also find out that issuing "umount" on the data store, reveals the ESX using it easier and faster then looking for mac addresses :-)

  5. Thanks the vsantraced worked for me thank you.

  6. Nice work, helped me with my migration.

  7. Cheers for the tips - after installing 5.5 and migrating vm from local to SAN, all that was left blocking datastore removal were the dump files and vsantraces.

  8. Thank you. Why can't VMware have a nice support article like this? Anyway, vsantraces worked well.

  9. Hi,
    well done helped alot.
    I additionaly had to change the Syslog directory on every Host.
    then it worked fine!

  10. It didn't helped me , after rebooting only issue fixed .

  11. Ran into this exact issue and your steps with the CoreDumps fixed it for me.

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Trackbacks and Pingbacks: