Issue

Installing or removing packages fails with "BOOTx64.EFI not found" error - ESXi 7.0

While preparing my ESXi Hosts for the recently released ESXi 7.0 Update 3, I came across a strange issue. When I try to install or remove any packages using esxcli, the update failed with the following error message:

[KeyError]
"filename 'var/db/payloads/boot_loader_efi/BOOTx64.EFI' not found"
Please refer to the log file for more details.

Any updates using Update Manager / Lifecycle Manager did also fail. ESXi was running at 7.0.2 build-17867351, which correlates to ESXi 7.0 Update 2a. Using "esxcli software profile list" reveals that this was a fresh installation and no packages have been updated yet. The image was customized by adding the vmkusb-nic-fling using PowerCLI Image Builder. After some troubleshooting, I was able to reproduce the Issue which helped me to identify the root cause and I also found a solution to make updates functional again.

Read More »Installing or removing packages fails with "BOOTx64.EFI not found" error - ESXi 7.0

Failed to load crypto64.efi - ESXi 7.0 U2 Upgrade Error

When you try to upgrade your ESXi host to the latest 7.0 U2 release using either the predefined update baselines or by using esxcli with the upgrade bundle, your ESXi host might fail to reboot with the following error message.

Loading /boot.cfg
Failed to load crypto64.efi
Fatal error: 15 (Not found)

The error can not be solved with the Shift+R method to restore the previous ESXi version. VMware is aware of the problem and has already removed the update bundle (VMware-ESXi-7.0U2-17630552-depot.zip) and Image Profile (ESXi-7.0.2-17630552-standard) from their repository. Currently, you only have two options to upgrade to ESXi 7.0 Update 2. If you already ran into the "Failed to load crypto64.efi" error, you have to take option 1, which will fix the error.

[Update 2021-03-13] - VMware has also disabled the image profile for 7.0.2. If you try an online update using ESXCLI or want to create a custom image using Imagebuilder, you get the following error:

[NoMatchError] No image profile found with name 'ESXi-7.0.2-17630552-standard' id = ESXi-7.0.2-17630552-standard Please refer to the log file for more details.

Read More »Failed to load crypto64.efi - ESXi 7.0 U2 Upgrade Error

Heads Up: VMFS6 Heap Exhaustion in ESXi 7.0

In ESXi 7.0 (Build 15843807) and 7.0b (Build 16324942), there is a known issue with the VMFS6 filesystem. The problem is solved in ESXi 7.0 Update 1. In certain workflows, memory is not freed correctly resulting in VMFS heap exhaustion. You might be affected when your system shows the following symptoms:

  • Datastores are showing "Not consumed" on hosts
  • Virtual Machines fail to vMotion
  • Virtual Machines become orphaned when powered off
  • Snapshot creation fails with "An error occurred while saving the snapshot: Error."

In the vmkernel.log, you see the following error messages:

  • Heap vmfs3 already at its maximum size. Cannot expand
  • Heap vmfs3: Maximum allowed growth (#) too small for size (#)
  • Failed to initialize VMFS distributed locking on volume #: Out of memory
  • Failed to get object 28 type 1 uuid # FD 0 gen 0: Out of memory

Read More »Heads Up: VMFS6 Heap Exhaustion in ESXi 7.0

VCSA 6.5 Broken Filesystem - "Welcome to Emergency Mode"

Today I managed to crash the storage used in my home lab. After fixing the FreeNAS box, my vCenter Server Appliance (Version 6.5 Update 1) refused to boot and after opening the console, it welcomed me with the following error message:

Welcome to emergency mode! After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to try again to boot into default mode.
Give root password for maintenance
(or press Control-D to continue):

Typically, this problem is caused by filesystem issues. This article explains how to fix the filesystem and get the appliance back up.

Read More »VCSA 6.5 Broken Filesystem - "Welcome to Emergency Mode"

Heads Up - ESXi not working on 7th Gen (Kaby Lake) Intel NUC7

I've received reports that the ESXi 6.5 and ESXi 6.0 installer fails to load on the latest 7th Gen NUCs:

  • NUC7i3BNH
  • NUC7i3BNK
  • NUC7i5BNH
  • NUC7i5BNK

The main issue is that the I219-V NIC is not recognized, so the installer fails with the well known "No Network Adapters" error message. Today I managed to get my hands on a NUC7i3BNH to narrow down the issue. By now I've not managed to get the embedded Network Adapter to work. A workaround with a USB-based NIC is possible.

[Update 2017-02-25 - A fix is available]

Read More »Heads Up - ESXi not working on 7th Gen (Kaby Lake) Intel NUC7

USB 3.0 devices detected as USB 2 in ESXi 6.0 and 5.5

In my latest post USB Devices as VMFS Datastore in vSphere ESXi 6.0 I had a problem with USB 3.0 devices that are detected as USB 2 in ESXi. I know that USB 3.0, also known as eXtensible Host Controller Interface (xHCI), is supported in ESXi 6.0 and ESXi 5.5 Build 2143827 or later. Unfortunately all of my devices are detected as USB 2.1, despite the USB 3 hub was visible. This problem applies to both, USB devices in path-through mode, and USB devices mounted from the command line with usbarbitrator disabled. The solution was quite simple and not related to an ESXi, but to a UEFI configuration.
xhci-smart-auto

Read More »USB 3.0 devices detected as USB 2 in ESXi 6.0 and 5.5

More Information on CVE-2015-5177 (ESXi OpenSLP Remote Code Execution)

You might be aware of the 3 critical security issues that VMware has published and fixed a couple of days ago in VMSA-2015-0007. The information provided in the security advisory regarding the first issue, CVE-2015-5177 (ESXi OpenSLP Remote Code Execution), are:

VMware ESXi contains a double free flaw in OpenSLP's SLPDProcessMessage() function. Exploitation of this issue may allow an unauthenticated attacker to remotely execute code on the ESXi host.

Relevant Releases
VMware ESXi 5.5 without patch ESXi550-201509101
VMware ESXi 5.1 without patch ESXi510-201510101
VMware ESXi 5.0 without patch ESXi500-201510101

In this post I am trying to give a better understanding of the vulnerability and its consequences. Please note that the information in this post are my personal opinions. I cannot guarantee that these information are accurate. The main fact is that VMware has published a fix and you should install the patch to be on the safe side. In the real world, you might have something like a "change process" where you can't rollout the patch for hundreds of systems immediately. Or you have a single ESXi that you don't want to reboot at the moment. In this situation, this post tries to help...

Read More »More Information on CVE-2015-5177 (ESXi OpenSLP Remote Code Execution)