vSphere with Kubernetes

Create Virtual Machines in vSphere with Tanzu using kubectl

This article explains how you can create Virtual Machines in Kubernetes Namespaces in vSphere with Tanzu. The deployment of Virtual Machines in Kubernetes namespaces using kubectl was shown in demonstrations but is currently (as of vSphere 7.0 U2) not supported. Only with third-party integrations like TKG, it is possible to create Virtual Machines by leveraging the vmoperator.

With the kubernetes-admin, accessible from the SupervisorControlPlane VM, you can create Virtual Machines today.

Please keep in mind that this is not officially supported by VMware.

Read More »Create Virtual Machines in vSphere with Tanzu using kubectl

vSphere with Tanzu - SupervisorControlPlaneVM Excessive Disk WRITE IO

After deploying the latest version of VMware vSphere with Tanzu (vCenter Server 7.0 U1d / v1.18.2-vsc0.0.7-17449972), I noticed that the Virtual Machines running the Control Plane (SupervisorControlPlaneVM) had a constant disk write IO of 15 MB/s with over 3000 IOPS. This was something I didn't see in previous versions and as this is a completely new setup with no namespaces created yet, there must be an issue.

After troubleshooting the Supervisor Control Plane, it turned out that the problem was caused by fluent-bit, which is the Log processor used by Kubernetes. The log was constantly spammed with debugging messages. Reducing the log level solved the problem for me.

[Update: 2021-03-14 - The problem is not resolved in vSphere 7.0 Update 2]

Read More »vSphere with Tanzu - SupervisorControlPlaneVM Excessive Disk WRITE IO

Tanzu Kubernetes Licensing in vSphere 7.0 Update 1

With the release of vSphere 7.0 Update 1, VMware introduced a new licensing model for its Tanzu Kubernetes integration. Basically, the licensing has been changed from an ESXi-Host license to a Cluster license that looks familiar to the vSAN license which is in place for a couple of years. The change does only affect how you have to apply the license. The entity to pay for is still a physical CPU.

In vSphere 7.0 GA, the license required to enable Kubernetes (aka. "Workload Management") was an add-on license for ESXi Hosts named "vSphere 7 Enterprise Plus with Kubernetes". With the introduction of vSphere 7.0 Update 1, which is also referred to as 7.0.1, "vSphere add-on for Kubernetes" has been rebranded and split into 4 licenses Tanzu Basic, Tanzu Standard, Tanzu

Read More »Tanzu Kubernetes Licensing in vSphere 7.0 Update 1

Quick Tip: Reset Tanzu SupervisorControlPlaneVM Alarms

When you are working with the Kubernetes Integration in vSphere 7.0, you might come into the situation where the SupervisorControlPlaneVM has an active alarm. Those Virtual Machines are deployed and controlled by the WCP Agent and even as an Administrator, you are not allowed to touch those objects.
You can't power then off, reboot, or migrate them using vMotion. The problem is that you can't even clear alarms. One alarm I recently had was the "vSphere HA virtual machine failover failed" alarm, which you usually see when the ESXi hostd crashed, but the Virtual Machines are still running.Read More »Quick Tip: Reset Tanzu SupervisorControlPlaneVM Alarms

Change TKG Cluster Service and Pod CIDR in Cloud Director 10.2

A major problem when deploying "vSphere with Tanzu" Clusters in VMware Cloud Director 10.2 is that the defaults for TKG Clusters are overlapping with the defaults for the Supervisor Cluster configured in vCenter Server during the Workload Management enablement.

When you deploy a Kubernetes Cluster using the new Container Extension in VCD 10.2, it deploys the cluster in a namespace on top of the Supervisor Cluster in the vCenter Server. The Supervisor Clusters IP address ranges for the Ingress CIDRs and Services CIDR must not overlap with IP addresses 10.96.0.0/12 and 192.168.0.0/16, which is the default for TKG Clusters. Unfortunately, 10.96.0.0 is also the default when enabling workload management so the deployment will fail when you stick to the defaults. The following error message is displayed when you have overlapping networks:

spec.settings.network.pods.cidrBlocks intersects with the network range of the external ip pools in network provider's configuration
spec.settings.network.pods.cidrBlocks intersects with the network range of the external ip pools LB in network provider's configuration

This article explains a workaround that you can apply when deleting and reconfiguring the Namespace Management with non-overlapping addresses is not an option.

Read More »Change TKG Cluster Service and Pod CIDR in Cloud Director 10.2

Troubleshooting "vSphere with Tanzu" Integration in VCD 10.2

During my first attempts to integrate "vSphere with Tanzu" into VMware Cloud Director 10.2, I had a couple of issues. The integration just wasn't as smooth as I expected and many configuration errors are not mitigated in the GUI. Also, there are a lot of prerequisites to strictly follow.

In this article, I'm going through the issues I had during the deployment and how to solve them.Read More »Troubleshooting "vSphere with Tanzu" Integration in VCD 10.2

Configure "vSphere with Tanzu" in VMware Cloud Director 10.2

With the release of Cloud Director 10.2, you can now integrate "vSphere with Tanzu" Kubernetes Clusters into VMware Cloud Director. That enabled you to create a self-service platform for Kubernetes Clusters that are backed by the Kubernetes integration in vSphere 7.0.

This article explains how to integrate vSphere with Tanzu in VMware Cloud Director 10.2

Read More »Configure "vSphere with Tanzu" in VMware Cloud Director 10.2

vSphere with Kubernetes - Which Supervisor Cluster Settings can be edited?

When you want to deploy Kubernetes on vSphere 7 it is crucial to plan the configuration thoroughly prior to enabling Workload Management. Many of the configuration parameters entered during the Workload Management wizard can not be changed after the deployment.

The following table show which settings can be changed after the initial deployment:

Read More »vSphere with Kubernetes - Which Supervisor Cluster Settings can be edited?

vSphere with Kubernetes Supports Multiple Tier-0 Gateways

During my first vSphere with Kubernetes tests, I had an issue where I was not able to activate Workload Management (Kubernetes) because it discovered multiple Tier-0 gateways. The configuration I used was vSphere 7.0 GA and an NSX-T 3.0 backed N-VDS. I had a previously configured Edge Cluster / Tier-0 Gateway for existing workloads and configured a new Edge Cluster / Tier-0 for Kubernetes.

In the Workload Management Wizard, no Cluster was compatible so I was forced to use the previously configured Tier-0 with some routing workarounds. The error message in wcpsvc.log stated "[...]has more than one tier0 gateway[...]".

Today I tried to find a solution and noticed that there was an update to the official Kubernetes Guide:

Read More »vSphere with Kubernetes Supports Multiple Tier-0 Gateways