While working with NSX-T, there are many reasons to access edge appliances using SSH. Most troubleshooting options are only available using nsxcli on the appliance itself. During the deployment, each appliance has 3 user account: root, admin, and audit. Alle Accounts are configured with password-based authentication. In a previous article, I've already described how to deploy SSH Keys using nsxcli, which allows a secure and comfortable authentication method. In this article, I'm explaining how to use ansible to deploy SSH public keys to NSX-T Edges. This option allows you to easily manage keys on a large platform.
When you try to connect an NSX-T based Segment to a virtual machine, the task fails with the following error message:
Reconfigure virtual machine - An error occurred during host configuration
In the nsx logfile on the ESXi host where the VM is located, the following error is displayed:
/var/log/nsx-syslog.log 2021-03-13T19:00:36Z nsx-opsagent: NSX 527252 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="527596" level="ERROR" errorCode="MPA44211"] [PortOp] Failed to create port 780b915d-1479-4eed-8e29-2364d9563f95 with VIF f3f605f2-38a1-4263-bbbd-81b189077f69 because DVS id is not found by transport-zone id 1b3a2f36-bfd1-443e-a0f6-4de01abc963e 2021-03-13T19:00:36Z nsx-opsagent: NSX 527252 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="527596" level="ERROR" errorCode="MPA42001"] [CreateLocalDvPort] createPort(uuid=780b915d-1479-4eed-8e29-2364d9563f95, zone=1b3a2f36-bfd1-443e-a0f6-4de01abc963e) failed: Failed to create port 780b915d-1479-4eed-8e29-2364d9563f95 with VIF f3f605f2-38a1-4263-bbbd-81b189077f69 because DVS id is not found by transport-zone id 1b3a2f36-bfd1-443e-a0f6-4de01abc963e
After deploying the latest version of VMware vSphere with Tanzu (vCenter Server 7.0 U1d / v1.18.2-vsc0.0.7-17449972), I noticed that the Virtual Machines running the Control Plane (SupervisorControlPlaneVM) had a constant disk write IO of 15 MB/s with over 3000 IOPS. This was something I didn't see in previous versions and as this is a completely new setup with no namespaces created yet, there must be an issue.
After troubleshooting the Supervisor Control Plane, it turned out that the problem was caused by fluent-bit, which is the Log processor used by Kubernetes. The log was constantly spammed with debugging messages. Reducing the log level solved the problem for me.
[Update: 2021-03-14 - The problem is not resolved in vSphere 7.0 Update 2]
In ESXi 7.0 (Build 15843807) and 7.0b (Build 16324942), there is a known issue with the VMFS6 filesystem. The problem is solved in ESXi 7.0 Update 1. In certain workflows, memory is not freed correctly resulting in VMFS heap exhaustion. You might be affected when your system shows the following symptoms:
- Datastores are showing "Not consumed" on hosts
- Virtual Machines fail to vMotion
- Virtual Machines become orphaned when powered off
- Snapshot creation fails with "An error occurred while saving the snapshot: Error."
In the vmkernel.log, you see the following error messages:
- Heap vmfs3 already at its maximum size. Cannot expand
- Heap vmfs3: Maximum allowed growth (#) too small for size (#)
- Failed to initialize VMFS distributed locking on volume #: Out of memory
- Failed to get object 28 type 1 uuid # FD 0 gen 0: Out of memory
When you are working with the Kubernetes Integration in vSphere 7.0, you might come into the situation where the SupervisorControlPlaneVM has an active alarm. Those Virtual Machines are deployed and controlled by the WCP Agent and even as an Administrator, you are not allowed to touch those objects.
You can't power then off, reboot, or migrate them using vMotion. The problem is that you can't even clear alarms. One alarm I recently had was the "vSphere HA virtual machine failover failed" alarm, which you usually see when the ESXi hostd crashed, but the Virtual Machines are still running.Read More »Quick Tip: Reset Tanzu SupervisorControlPlaneVM Alarms
Visual Studio Code Error "No match was found for the specified search criteria and module name PackageManagement"
Visual Studio Code asks to perform an update when launching the PowerShell Integrated Console. The Update fails with the following error message:
PS> powershell.exe -NoLogo -NoProfile -Command 'Install-Module -Name PackageManagement -Force -MinimumVersion 1.4.6 -Scope CurrentUser -AllowClobber'
PackageManagement\Find-Package : No match was found for the specified search criteria and module name 'PackageManagement'. Try Get-PSRepository to see all available registered module repositories.
Get-PSRepository returns the following error:Read More »Visual Studio Code Error "No match was found for the specified search criteria and module name PackageManagement"
When you try to install ESXi 7.0 with a USB NIC only, the installation fails at 81% with the following error message:
Exception: No vmknic tagged for management was found.
Some homelab systems like the Intel 10th Gen NUC are not equipped with a compatible network adapter. As a workaround, you can use a USB NIC and create a customized image to install ESXi. The installation fails as the ESXi installer can't assign the USB NIC as a management adapter because it specifically searches for a "vmnic#", not "vusb#" adapter.
This article explains how to proceed with the 81% installation error and get the system to work.
When you try to install a VMware ESXi Update using esxcli, the upgrade fails with the following error message:
"[Errno 28] No space left on device"
The problem is caused by ESXi not having enough free space available to extract the installation packages. This article explains how to solve the issue by enabling swapping to a Datastore.
When you deploy a Virtual Machine from OVF/OVA you can sometimes preconfigure passwords using OVF properties. All configuration parameters set during the deployment can be viewed later in the Web Client within VM > Configure > Settings > vApp Options but when you want to retrieve the password, the actual value is hidden:
You can't access the password from the Client, MOB, or using the API. This article explains how you can retrieve vApp option passwords from the vCenter Database.
This article explains how to prepare the vCenter Server Appliance to connect with external Postgres Management Tools like pgAdmin. This method works with vCenter Server Appliance version 6.5, 6.7, and 7.0.