Issue

VMware ESXi 7.0 Update 3 on Intel NUC

VMware vSphere ESXi 7.0 Update 3 has been released in October and before you start to deploy it to production, you want to evaluate it in your testing environment or homelab. If you have Intel NUCs or similar hardware you should be very careful when updating to new ESXi releases as there might be issues. Please always keep in mind that this is not an officially supported platform and there might be compatibility issues.

In vSphere 7.0, there are ups and downs with consumer-grade network adapters. Since the deprecation of VMKlinux drivers, there is no option to use Realtek-based NICs, and previous versions had problems with the ne1000 driver. Luckily there is the great Community Networking Driver for ESXi Fling that adds support for a bunch of network cards and VMKUSB-NIC-FLING always covers your back.

I've updated my NUC portfolio to check which NUCs are safe to update and what considerations you have to take before installing the update. Additionally, I'm taking a look at the consequences of the recently deprecated USB/SD-Card usage for ESXi Installations and some general Issues in 7.0u3.

Read More »VMware ESXi 7.0 Update 3 on Intel NUC

Installing or removing packages fails with "BOOTx64.EFI not found" error - ESXi 7.0

While preparing my ESXi Hosts for the recently released ESXi 7.0 Update 3, I came across a strange issue. When I try to install or remove any packages using esxcli, the update failed with the following error message:

[KeyError]
"filename 'var/db/payloads/boot_loader_efi/BOOTx64.EFI' not found"
Please refer to the log file for more details.

Any updates using Update Manager / Lifecycle Manager did also fail. ESXi was running at 7.0.2 build-17867351, which correlates to ESXi 7.0 Update 2a. Using "esxcli software profile list" reveals that this was a fresh installation and no packages have been updated yet. The image was customized by adding the vmkusb-nic-fling using PowerCLI Image Builder. After some troubleshooting, I was able to reproduce the Issue which helped me to identify the root cause and I also found a solution to make updates functional again.

Read More »Installing or removing packages fails with "BOOTx64.EFI not found" error - ESXi 7.0

Failed to load crypto64.efi - ESXi 7.0 U2 Upgrade Error

When you try to upgrade your ESXi host to the latest 7.0 U2 release using either the predefined update baselines or by using esxcli with the upgrade bundle, your ESXi host might fail to reboot with the following error message.

Loading /boot.cfg
Failed to load crypto64.efi
Fatal error: 15 (Not found)

The error can not be solved with the Shift+R method to restore the previous ESXi version. VMware is aware of the problem and has already removed the update bundle (VMware-ESXi-7.0U2-17630552-depot.zip) and Image Profile (ESXi-7.0.2-17630552-standard) from their repository. Currently, you only have two options to upgrade to ESXi 7.0 Update 2. If you already ran into the "Failed to load crypto64.efi" error, you have to take option 1, which will fix the error.

[Update 2021-03-13] - VMware has also disabled the image profile for 7.0.2. If you try an online update using ESXCLI or want to create a custom image using Imagebuilder, you get the following error:

[NoMatchError] No image profile found with name 'ESXi-7.0.2-17630552-standard' id = ESXi-7.0.2-17630552-standard Please refer to the log file for more details.

Read More »Failed to load crypto64.efi - ESXi 7.0 U2 Upgrade Error

Heads Up: VMFS6 Heap Exhaustion in ESXi 7.0

In ESXi 7.0 (Build 15843807) and 7.0b (Build 16324942), there is a known issue with the VMFS6 filesystem. The problem is solved in ESXi 7.0 Update 1. In certain workflows, memory is not freed correctly resulting in VMFS heap exhaustion. You might be affected when your system shows the following symptoms:

  • Datastores are showing "Not consumed" on hosts
  • Virtual Machines fail to vMotion
  • Virtual Machines become orphaned when powered off
  • Snapshot creation fails with "An error occurred while saving the snapshot: Error."

In the vmkernel.log, you see the following error messages:

  • Heap vmfs3 already at its maximum size. Cannot expand
  • Heap vmfs3: Maximum allowed growth (#) too small for size (#)
  • Failed to initialize VMFS distributed locking on volume #: Out of memory
  • Failed to get object 28 type 1 uuid # FD 0 gen 0: Out of memory

Read More »Heads Up: VMFS6 Heap Exhaustion in ESXi 7.0

VCSA 6.5 Broken Filesystem - "Welcome to Emergency Mode"

Today I managed to crash the storage used in my home lab. After fixing the FreeNAS box, my vCenter Server Appliance (Version 6.5 Update 1) refused to boot and after opening the console, it welcomed me with the following error message:

Welcome to emergency mode! After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to try again to boot into default mode.
Give root password for maintenance
(or press Control-D to continue):

Typically, this problem is caused by filesystem issues. This article explains how to fix the filesystem and get the appliance back up.

Read More »VCSA 6.5 Broken Filesystem - "Welcome to Emergency Mode"

Heads Up - ESXi not working on 7th Gen (Kaby Lake) Intel NUC7

I've received reports that the ESXi 6.5 and ESXi 6.0 installer fails to load on the latest 7th Gen NUCs:

  • NUC7i3BNH
  • NUC7i3BNK
  • NUC7i5BNH
  • NUC7i5BNK

The main issue is that the I219-V NIC is not recognized, so the installer fails with the well known "No Network Adapters" error message. Today I managed to get my hands on a NUC7i3BNH to narrow down the issue. By now I've not managed to get the embedded Network Adapter to work. A workaround with a USB-based NIC is possible.

[Update 2017-02-25 - A fix is available]

Read More »Heads Up - ESXi not working on 7th Gen (Kaby Lake) Intel NUC7

USB 3.0 devices detected as USB 2 in ESXi 6.0 and 5.5

In my latest post USB Devices as VMFS Datastore in vSphere ESXi 6.0 I had a problem with USB 3.0 devices that are detected as USB 2 in ESXi. I know that USB 3.0, also known as eXtensible Host Controller Interface (xHCI), is supported in ESXi 6.0 and ESXi 5.5 Build 2143827 or later. Unfortunately all of my devices are detected as USB 2.1, despite the USB 3 hub was visible. This problem applies to both, USB devices in path-through mode, and USB devices mounted from the command line with usbarbitrator disabled. The solution was quite simple and not related to an ESXi, but to a UEFI configuration.
xhci-smart-auto

Read More »USB 3.0 devices detected as USB 2 in ESXi 6.0 and 5.5