With the release of vSphere 7.0 Update 2, a new version of the vSphere authentication plugin for kubectl has been released. The new plugin, which can be downloaded from the Supervisor Control Plane after enabling Workload Management, has a neat new feature that allows you to save the password in an environment variable.
This article explains how you can create Virtual Machines in Kubernetes Namespaces in vSphere with Tanzu. The deployment of Virtual Machines in Kubernetes namespaces using kubectl was shown in demonstrations but is currently (as of vSphere 7.0 U2) not supported. Only with third-party integrations like TKG, it is possible to create Virtual Machines by leveraging the vmoperator.
With the kubernetes-admin, accessible from the SupervisorControlPlane VM, you can create Virtual Machines today.
Please keep in mind that this is not officially supported by VMware.
While working with NSX-T, there are many reasons to access edge appliances using SSH. Most troubleshooting options are only available using nsxcli on the appliance itself. During the deployment, each appliance has 3 user account: root, admin, and audit. Alle Accounts are configured with password-based authentication. In a previous article, I've already described how to deploy SSH Keys using nsxcli, which allows a secure and comfortable authentication method. In this article, I'm explaining how to use ansible to deploy SSH public keys to NSX-T Edges. This option allows you to easily manage keys on a large platform.
VMware vSphere ESXi 7.0 Update 2 has been released this week and before you start to deploy it to production, you want to evaluate it in your testing environment or homelab. If you have Intel NUCs you should always be very careful when updating to new ESXi releases as there might be issues. Please always keep in mind that this is not an officially supported platform.
Within the 7.0 releases, there are many issues with consumer network adapters, like the deprecation of VMKlinux drivers and thus the missing support for Realtek NICs, and the up and downs with the ne1000 driver.
To be on the safe side, I'm doing a quick checkup on which NUCs are safe to update and what considerations you have to take before installing the update. Also, I'm quickly explaining the options to workaround the crypto64.efi issue.
When you try to connect an NSX-T based Segment to a virtual machine, the task fails with the following error message:
Reconfigure virtual machine - An error occurred during host configuration
In the nsx logfile on the ESXi host where the VM is located, the following error is displayed:
/var/log/nsx-syslog.log 2021-03-13T19:00:36Z nsx-opsagent: NSX 527252 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="527596" level="ERROR" errorCode="MPA44211"] [PortOp] Failed to create port 780b915d-1479-4eed-8e29-2364d9563f95 with VIF f3f605f2-38a1-4263-bbbd-81b189077f69 because DVS id is not found by transport-zone id 1b3a2f36-bfd1-443e-a0f6-4de01abc963e 2021-03-13T19:00:36Z nsx-opsagent: NSX 527252 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="527596" level="ERROR" errorCode="MPA42001"] [CreateLocalDvPort] createPort(uuid=780b915d-1479-4eed-8e29-2364d9563f95, zone=1b3a2f36-bfd1-443e-a0f6-4de01abc963e) failed: Failed to create port 780b915d-1479-4eed-8e29-2364d9563f95 with VIF f3f605f2-38a1-4263-bbbd-81b189077f69 because DVS id is not found by transport-zone id 1b3a2f36-bfd1-443e-a0f6-4de01abc963e
When you try to upgrade your ESXi host to the latest 7.0 U2 release using either the predefined update baselines or by using esxcli with the upgrade bundle, your ESXi host might fail to reboot with the following error message.
Failed to load crypto64.efi
Fatal error: 15 (Not found)
The error can not be solved with the Shift+R method to restore the previous ESXi version. VMware is aware of the problem and has already removed the update bundle (VMware-ESXi-7.0U2-17630552-depot.zip) and Image Profile (ESXi-7.0.2-17630552-standard) from their repository. Currently, you only have two options to upgrade to ESXi 7.0 Update 2. If you already ran into the "Failed to load crypto64.efi" error, you have to take option 1, which will fix the error.
[Update 2021-03-13] - VMware has also disabled the image profile for 7.0.2. If you try an online update using ESXCLI or want to create a custom image using Imagebuilder, you get the following error:
[NoMatchError] No image profile found with name 'ESXi-7.0.2-17630552-standard' id = ESXi-7.0.2-17630552-standard Please refer to the log file for more details.
When you've configured automated backups in NSX-T, you might be unaware that failed backup jobs do not trigger alarms in the integrated NSX-T alarm dashboard. When a backup fails, you can only see the following error message in the Backup & Restore configuration:
At the moment, you have to manually check that the backup is running as expected. This can also be done using the API:
NSX-T is a critical infrastructure component and it is crucial to have a working backup and restore plan. With complex products, the backup and restore strategy gets more complicated. When working with Virtual Machines, the backup is usually done with VMware Snapshots, which is super convenient. Unfortunately, with the complexity of NSX-T which has many components like clustered Managers, Transport Nodes, and ESXi Kernel Modules, you can't use snapshots as a backup strategy.
This article provides an overview of how to backup NSX-T, and how the restore is done properly.
After deploying the latest version of VMware vSphere with Tanzu (vCenter Server 7.0 U1d / v1.18.2-vsc0.0.7-17449972), I noticed that the Virtual Machines running the Control Plane (SupervisorControlPlaneVM) had a constant disk write IO of 15 MB/s with over 3000 IOPS. This was something I didn't see in previous versions and as this is a completely new setup with no namespaces created yet, there must be an issue.
After troubleshooting the Supervisor Control Plane, it turned out that the problem was caused by fluent-bit, which is the Log processor used by Kubernetes. The log was constantly spammed with debugging messages. Reducing the log level solved the problem for me.
[Update: 2021-03-14 - The problem is not resolved in vSphere 7.0 Update 2]
Intel has finally announced their 11th Generation NUCs. For the first time, all three product lines are announced at the same time. The NUC series is very popular to be used in homelabs or for running VMware ESXi. They are small, silent, transportable, and have very low power consumption.
- Enthusiast (Phantom Canyon) - Successor to the 8th Gen Hades Canyon
- Pro (Tiger Canyon) - Successor to the 8th Gen Provo Canyon
- Performance (Panther Canyon) - Successor to the 10th Gen Frost Canyon
In this article, I'm going to take a look at the 3 different product lines and how they compare to each other and previous NUCs.