Tag Archives: Troubleshooting

Heads Up: VMFS6 Heap Exhaustion in ESXi 7.0

In ESXi 7.0 (Build 15843807) and 7.0b (Build 16324942), there is a known issue with the VMFS6 filesystem. The problem is solved in ESXi 7.0 Update 1. In certain workflows, memory is not freed correctly resulting in VMFS heap exhaustion. You might be affected when your system shows the following symptoms:

  • Datastores are showing "Not consumed" on hosts
  • Virtual Machines fail to vMotion
  • Virtual Machines become orphaned when powered off
  • Snapshot creation fails with "An error occurred while saving the snapshot: Error."

In the vmkernel.log, you see the following error messages:

  • Heap vmfs3 already at its maximum size. Cannot expand
  • Heap vmfs3: Maximum allowed growth (#) too small for size (#)
  • Failed to initialize VMFS distributed locking on volume #: Out of memory
  • Failed to get object 28 type 1 uuid # FD 0 gen 0: Out of memory

Read more »

Quick Tip: Reset Tanzu SupervisorControlPlaneVM Alarms

When you are working with the Kubernetes Integration in vSphere 7.0, you might come into the situation where the SupervisorControlPlaneVM has an active alarm. Those Virtual Machines are deployed and controlled by the WCP Agent and even as an Administrator, you are not allowed to touch those objects.
You can't power then off, reboot, or migrate them using vMotion. The problem is that you can't even clear alarms. One alarm I recently had was the "vSphere HA virtual machine failover failed" alarm, which you usually see when the ESXi hostd crashed, but the Virtual Machines are still running. Read more »

Visual Studio Code Error "No match was found for the specified search criteria and module name PackageManagement"

Visual Studio Code asks to perform an update when launching the PowerShell Integrated Console. The Update fails with the following error message:

PS> powershell.exe -NoLogo -NoProfile -Command 'Install-Module -Name PackageManagement -Force -MinimumVersion 1.4.6 -Scope CurrentUser -AllowClobber'

PackageManagement\Find-Package : No match was found for the specified search criteria and module name 'PackageManagement'. Try Get-PSRepository to see all available registered module repositories.

Get-PSRepository returns the following error: Read more »

Solution: ESXi Installation with USB NIC only fails at 81%

When you try to install ESXi 7.0 with a USB NIC only, the installation fails at 81% with the following error message:

Exception: No vmknic tagged for management was found.


Some homelab systems like the Intel 10th Gen NUC are not equipped with a compatible network adapter. As a workaround, you can use a USB NIC and create a customized image to install ESXi. The installation fails as the ESXi installer can't assign the USB NIC as a management adapter because it specifically searches for a "vmnic#", not "vusb#" adapter.

This article explains how to proceed with the 81% installation error and get the system to work.

Read more »

ESXi Update Error "[Errno 28] No space left on device"

When you try to install a VMware ESXi Update using esxcli, the upgrade fails with the following error message:

"[Errno 28] No space left on device"

The problem is caused by ESXi not having enough free space available to extract the installation packages. This article explains how to solve the issue by enabling swapping to a Datastore.

Read more »

Retrieve VMware Virtual Machine Password from OVF properties

When you deploy a Virtual Machine from OVF/OVA you can sometimes preconfigure passwords using OVF properties. All configuration parameters set during the deployment can be viewed later in the Web Client within VM > Configure > Settings > vApp Options but when you want to retrieve the password, the actual value is hidden:
You can't access the password from the Client, MOB, or using the API. This article explains how you can retrieve vApp option passwords from the vCenter Database.

Read more »

Manage vCSA Embedded vPostgres Database with pgAdmin

This article explains how to prepare the vCenter Server Appliance to connect with external Postgres Management Tools like pgAdmin. This method works with vCenter Server Appliance version 6.5, 6.7, and 7.0.

Read more »

SCSI Sense Code Decoder OP Code Update

I've added a new feature to my online SCSI Sense Code Decoder. It is now possible to enter the Command that has been reported to be failed into the form. The command is displayed in the error message and can be important for troubleshooting:

The Command can be optionally entered in the form and will be translated:

Another small fix: Previously, Host Status and Sense Key fields were case sensitive and have to be entered in lower case. This requirement has been removed, you can now use lower or upper case letters.

The next feature I'm working on is to make the decoder to work with scripts to allow even better and faster troubleshooting. Any other feature requests are welcome.

VMware ESXi SCSI Sense Code Decoder

Ping from specific VMkernel adapter in vSphere 6

When troubleshooting network problems on ESXi hosts you want to specify the outgoing VMkernel adapter. As explained here you can ping from a specific VMkernel adapter with the -I parameter. In vSphere 6.0, or with VXLAN activated, this might not work as expected and displays the following error.

[root@esx:~] ping -I vmk1 10.1.1.1
Unknown interface 'vmk1': Invalid argument

The problem is related to the multiple TCP/IP Stack features introduced in vSphere 6.0. To ping from specific VMkernel adapters that are not in the default Stack (defaultTcpipStack) you have to manually specify the NetStack with the -S parameter.

Read more »

How long are Virtual Machines stunned for Snapshots and vMotion?

I've read this excellent article by Cormac Hogan explaining why and when a Virtual Machine receives a "stun". This post is a follow-up explaining how you can determine the duration of a Virtual Machine stun on the most common vSphere functions:

  • Create a Snapshot
  • Delete a Snapshot
  • vMotion

Read more »