Troubleshooting

Deploy NSX-T Edge VM SSH Keys with Ansible

While working with NSX-T, there are many reasons to access edge appliances using SSH. Most troubleshooting options are only available using nsxcli on the appliance itself. During the deployment, each appliance has 3 user account: root, admin, and audit. Alle Accounts are configured with password-based authentication. In a previous article, I've already described how to deploy SSH Keys using nsxcli, which allows a secure and comfortable authentication method. In this article, I'm explaining how to use ansible to deploy SSH public keys to NSX-T Edges. This option allows you to easily manage keys on a large platform.

Read More »Deploy NSX-T Edge VM SSH Keys with Ansible

Error when connecting Virtual Machine to NSX-T Segments

When you try to connect an NSX-T based Segment to a virtual machine, the task fails with the following error message:

Reconfigure virtual machine - An error occurred during host configuration

In the nsx logfile on the ESXi host where the VM is located, the following error is displayed:

/var/log/nsx-syslog.log
2021-03-13T19:00:36Z nsx-opsagent[527252]: NSX 527252 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="527596" level="ERROR" errorCode="MPA44211"] [PortOp] Failed to create port 780b915d-1479-4eed-8e29-2364d9563f95 with VIF f3f605f2-38a1-4263-bbbd-81b189077f69 because DVS id is not found by transport-zone id 1b3a2f36-bfd1-443e-a0f6-4de01abc963e
2021-03-13T19:00:36Z nsx-opsagent[527252]: NSX 527252 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="527596" level="ERROR" errorCode="MPA42001"] [CreateLocalDvPort] createPort(uuid=780b915d-1479-4eed-8e29-2364d9563f95, zone=1b3a2f36-bfd1-443e-a0f6-4de01abc963e) failed: Failed to create port 780b915d-1479-4eed-8e29-2364d9563f95 with VIF f3f605f2-38a1-4263-bbbd-81b189077f69 because DVS id is not found by transport-zone id 1b3a2f36-bfd1-443e-a0f6-4de01abc963e

 

Read More »Error when connecting Virtual Machine to NSX-T Segments

vSphere with Tanzu - SupervisorControlPlaneVM Excessive Disk WRITE IO

After deploying the latest version of VMware vSphere with Tanzu (vCenter Server 7.0 U1d / v1.18.2-vsc0.0.7-17449972), I noticed that the Virtual Machines running the Control Plane (SupervisorControlPlaneVM) had a constant disk write IO of 15 MB/s with over 3000 IOPS. This was something I didn't see in previous versions and as this is a completely new setup with no namespaces created yet, there must be an issue.

After troubleshooting the Supervisor Control Plane, it turned out that the problem was caused by fluent-bit, which is the Log processor used by Kubernetes. The log was constantly spammed with debugging messages. Reducing the log level solved the problem for me.

[Update: 2021-03-14 - The problem is not resolved in vSphere 7.0 Update 2]

Read More »vSphere with Tanzu - SupervisorControlPlaneVM Excessive Disk WRITE IO

Heads Up: VMFS6 Heap Exhaustion in ESXi 7.0

In ESXi 7.0 (Build 15843807) and 7.0b (Build 16324942), there is a known issue with the VMFS6 filesystem. The problem is solved in ESXi 7.0 Update 1. In certain workflows, memory is not freed correctly resulting in VMFS heap exhaustion. You might be affected when your system shows the following symptoms:

  • Datastores are showing "Not consumed" on hosts
  • Virtual Machines fail to vMotion
  • Virtual Machines become orphaned when powered off
  • Snapshot creation fails with "An error occurred while saving the snapshot: Error."

In the vmkernel.log, you see the following error messages:

  • Heap vmfs3 already at its maximum size. Cannot expand
  • Heap vmfs3: Maximum allowed growth (#) too small for size (#)
  • Failed to initialize VMFS distributed locking on volume #: Out of memory
  • Failed to get object 28 type 1 uuid # FD 0 gen 0: Out of memory

Read More »Heads Up: VMFS6 Heap Exhaustion in ESXi 7.0

Quick Tip: Reset Tanzu SupervisorControlPlaneVM Alarms

When you are working with the Kubernetes Integration in vSphere 7.0, you might come into the situation where the SupervisorControlPlaneVM has an active alarm. Those Virtual Machines are deployed and controlled by the WCP Agent and even as an Administrator, you are not allowed to touch those objects.
You can't power then off, reboot, or migrate them using vMotion. The problem is that you can't even clear alarms. One alarm I recently had was the "vSphere HA virtual machine failover failed" alarm, which you usually see when the ESXi hostd crashed, but the Virtual Machines are still running.Read More »Quick Tip: Reset Tanzu SupervisorControlPlaneVM Alarms

Visual Studio Code Error "No match was found for the specified search criteria and module name PackageManagement"

Visual Studio Code asks to perform an update when launching the PowerShell Integrated Console. The Update fails with the following error message:

PS> powershell.exe -NoLogo -NoProfile -Command 'Install-Module -Name PackageManagement -Force -MinimumVersion 1.4.6 -Scope CurrentUser -AllowClobber'

PackageManagement\Find-Package : No match was found for the specified search criteria and module name 'PackageManagement'. Try Get-PSRepository to see all available registered module repositories.

Get-PSRepository returns the following error:Read More »Visual Studio Code Error "No match was found for the specified search criteria and module name PackageManagement"

Solution: ESXi Installation with USB NIC only fails at 81%

When you try to install ESXi 7.0 with a USB NIC only, the installation fails at 81% with the following error message:

Exception: No vmknic tagged for management was found.


Some homelab systems like the Intel 10th Gen NUC are not equipped with a compatible network adapter. As a workaround, you can use a USB NIC and create a customized image to install ESXi. The installation fails as the ESXi installer can't assign the USB NIC as a management adapter because it specifically searches for a "vmnic#", not "vusb#" adapter.

This article explains how to proceed with the 81% installation error and get the system to work.

Read More »Solution: ESXi Installation with USB NIC only fails at 81%

Retrieve VMware Virtual Machine Password from OVF properties

When you deploy a Virtual Machine from OVF/OVA you can sometimes preconfigure passwords using OVF properties. All configuration parameters set during the deployment can be viewed later in the Web Client within VM > Configure > Settings > vApp Options but when you want to retrieve the password, the actual value is hidden:
You can't access the password from the Client, MOB, or using the API. This article explains how you can retrieve vApp option passwords from the vCenter Database.

Read More »Retrieve VMware Virtual Machine Password from OVF properties