Installing or removing packages fails with "BOOTx64.EFI not found" error - ESXi 7.0

While preparing my ESXi Hosts for the recently released ESXi 7.0 Update 3, I came across a strange issue. When I try to install or remove any packages using esxcli, the update failed with the following error message:

[KeyError]
"filename 'var/db/payloads/boot_loader_efi/BOOTx64.EFI' not found"
Please refer to the log file for more details.

Any updates using Update Manager / Lifecycle Manager did also fail. ESXi was running at 7.0.2 build-17867351, which correlates to ESXi 7.0 Update 2a. Using "esxcli software profile list" reveals that this was a fresh installation and no packages have been updated yet. The image was customized by adding the vmkusb-nic-fling using PowerCLI Image Builder. After some troubleshooting, I was able to reproduce the Issue which helped me to identify the root cause and I also found a solution to make updates functional again.

Cause for "BOOTx64.EFI not found" Errors

As mentioned in the error message, a file named "var/db/payloads/boot_loader_efi/BOOTx64.EFI" is missing. Filenames with relative paths are usually used in tar archives, so I suspect the issue to be located in corrupted or missing boot files. However, booting and rebooting the ESXi hosts works without issues. After checking the filesystem, I found out that the file is supposed to be provided by basemisc.tgz, located in the /bootbank/ directory. This is the content on a broken ESXi Host:

# tar -tvf /bootbank/basemisc.tgz
drwxr-xr-x 0/0         0 1970-01-01 00:00:00 var/
drwxr-xr-x 0/0         0 1970-01-01 00:00:00 var/db/
drwxr-xr-x 0/0         0 1970-01-01 00:00:00 var/db/payloads/
drwxr-xr-x 0/0         0 1970-01-01 00:00:00 var/db/payloads/boot/
-rw-r--r-- 0/0        21 1970-01-01 00:00:00 var/db/payloads/boot/useropts

And this is what it is supposed to look like:

# tar -tvf /bootbank/basemisc.tgz
drwxr-xr-x 0/0       0 1970-01-01 01:00 var/
drwxr-xr-x 0/0       0 1970-01-01 01:00 var/db/
drwxr-xr-x 0/0       0 1970-01-01 01:00 var/db/payloads/
drwxr-xr-x 0/0       0 1970-01-01 01:00 var/db/payloads/boot_loader_efi/
-rw-r--r-- 0/0  169320 1970-01-01 01:00 var/db/payloads/boot_loader_efi/BOOTx64.EFI
drwxr-xr-x 0/0       0 1970-01-01 01:00 var/db/payloads/text/
-rw-r--r-- 0/0    2202 1970-01-01 01:00 var/db/payloads/text/README
drwxr-xr-x 0/0       0 1970-01-01 01:00 var/db/payloads/eltorito_image_efi/
-rw-r--r-- 0/0 1048576 1970-01-01 01:00 var/db/payloads/eltorito_image_efi/efiboot.img
drwxr-xr-x 0/0       0 1970-01-01 01:00 var/db/payloads/upgrade/
-rw-r--r-- 0/0 1970127 1970-01-01 01:00 var/db/payloads/upgrade/esximage.zip
-rw-r--r-- 0/0   64011 1970-01-01 01:00 var/db/payloads/upgrade/extlinux
drwxr-xr-x 0/0       0 1970-01-01 01:00 var/db/payloads/boot_com32_bios/
-rw-r--r-- 0/0     512 1970-01-01 01:00 var/db/payloads/boot_com32_bios/fatBootSector
-rw-r--r-- 0/0     424 1970-01-01 01:00 var/db/payloads/upgrade/gptmbr.bin
drwxr-xr-x 0/0       0 1970-01-01 01:00 var/db/payloads/boot_iso_bios/
-rw-r--r-- 0/0   14336 1970-01-01 01:00 var/db/payloads/boot_iso_bios/isolinux.bin
-rw-r--r-- 0/0   15218 1970-01-01 01:00 var/db/payloads/boot_com32_bios/ldlinux.sys
-rw-r--r-- 0/0   95372 1970-01-01 01:00 var/db/payloads/boot_com32_bios/mboot.c32
-rw-r--r-- 0/0     440 1970-01-01 01:00 var/db/payloads/upgrade/mbr.bin
-rw-r--r-- 0/0   53456 1970-01-01 01:00 var/db/payloads/boot_com32_bios/menu.c32
-rw-r--r-- 0/0    1696 1970-01-01 01:00 var/db/payloads/upgrade/metadata.xml
-rw-r--r-- 0/0 7429886 1970-01-01 01:00 var/db/payloads/text/osl.txt
-rw-r--r-- 0/0  150014 1970-01-01 01:00 var/db/payloads/upgrade/precheck.py
-rw-r--r-- 0/0    5493 1970-01-01 01:00 var/db/payloads/upgrade/prep.py
-rw-r--r-- 0/0   62292 1970-01-01 01:00 var/db/payloads/boot_com32_bios/safeboot.c32
-rw-r--r-- 0/0   92936 1970-01-01 01:00 var/db/payloads/boot_loader_efi/safeboot.efi

To fix the issue, you have to replace the file with a fixed version. You can simply copy the file from an ESXi ISO, or a working ESXi Installation. The source needs to have the same build number!

# vmware -v
VMware ESXi 7.0.2 build-17867351

Mount an ISO containing ESXi 7.0.2 build 17867351 and copy BASEMISC.TGZ. If you do not find the file, you have a broken ISO! Alternatively, with a working ESXi, you can also copy /bootbank/basemisc.tgz from there. Make sure to double-check the size! It should be about 1.8 MB. If you do not have a working ESXi or ISO you have to create a custom ISO. (Make sure to read "What causes broken ESX Images?" for instructions!)

Copy the file onto the broken ESXi host using scp (Or GUI tools like WinSCP).

SSH to the ESXi host and overwrite the broken basemisc.tgz (File on ISO is all uppercase, file in /bootbank/ is all lowercase!)

# mv BASEMISC.TGZ /bootbank/basemisc.tgz
mv: overwrite '/bootbank/basemisc.tgz'? yes

Reboot the ESXi.

You should now be able to install or remove packages. (Here: remove vmkusb-nuc-fling, which does not work with 7.0u3 yet)

 

What causes broken ESX Images?

The image I used to install ESXi was customized (Adding vmkusb-nic-fling) using PowerCLI Image Builder. After creating a new Image, I figured out that despite I used exactly the same method, the outcome is different:

The diff reveals that at least BASEMISC.TGZ is just missing in the old ISO.

# diff -r ESXi-7.0U2a-USBNIC ESXi-7.0U2a-USBNIC-NEW
Only in ESXi-7.0U2a-USBNIC-NEW: BASEMISC.TGZ
Binary files ESXi-7.0U2a-USBNIC/[BOOT]/1-Boot-NoEmul.img and ESXi-7.0U2a-USBNIC-NEW/[BOOT]/1-Boot-NoEmul.img differ
Binary files ESXi-7.0U2a-USBNIC/BOOT.CAT and ESXi-7.0U2a-USBNIC-NEW/BOOT.CAT differ
Binary files ESXi-7.0U2a-USBNIC/IMGDB.TGZ and ESXi-7.0U2a-USBNIC-NEW/IMGDB.TGZ differ
Binary files ESXi-7.0U2a-USBNIC/ISOLINUX.BIN and ESXi-7.0U2a-USBNIC-NEW/ISOLINUX.BIN differ
Only in ESXi-7.0U2a-USBNIC-NEW: RESVIBS.TGZ
Binary files ESXi-7.0U2a-USBNIC/UPGRADE/METADATA.ZIP and ESXi-7.0U2a-USBNIC-NEW/UPGRADE/METADATA.ZIP differ
diff -r ESXi-7.0U2a-USBNIC/UPGRADE/PROFILE.XML ESXi-7.0U2a-USBNIC-NEW/UPGRADE/PROFILE.XML

After creating dozens of ISOs, I got to the root cause: Old Versions of Image Build create those broken Images!

Conclusion: Make sure to update the VMware.PowerCLI Module when customizing Images!

The issue is resolved in version 7.0.2.17849781

> Get-Module VMware.ImageBuilder

ModuleType Version        Name                ExportedCommands
---------- -------        ----                ----------------
Script     7.0.2.17849781 VMware.ImageBuilder {Add-EsxSoftwareDepot, Add-EsxSoftwarePackage, Compare-EsxImageProfile, Export-EsxImageProfile...}

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.