VMware vSphere with Kubernetes Guide Part 6 - Tanzu Kubernetes Cluster

This is Part 6 of my "VMware vSphere with Kubernetes" Guide. In this article, I'm going to deploy a Tanzu Kubernetes Cluster (TKC). A TKC is a fully-featured version of the open-source Kubernetes container platform. You can provision and operate Tanzu Kubernetes clusters on top of the Supervisor Cluster.

What is a Tanzu Kubernetes Cluster?

A Tanzu Kubernetes Cluster is a fully-featured version of the open-source Kubernetes container platform than can be provided on top of the Supervisor Cluster. When deploying workloads, you can either deploy them directly to a Supervisor Clusters Namespace or inside a TKC running in the Namespace.

What are the advantages when running a TKC?

Use vSphere Pods running in the Supervisor Cluster when you want to:

  • Run containers without needing to customize a Kubernetes cluster.
  • Create containerized applications with strong resource and security isolation.
  • Deploy vSphere Pods directly on ESXi hosts.

Use a Tanzu Kubernetes Cluster when you want to:

  • Run containerized applications on open-source Kubernetes software.
  • Customize the Kubernetes cluster, including root level access to control plane and
    worker nodes.
  • Use different Kubernetes versions without requiring infrastructure upgrades.
  • Create Kubernetes namespaces using the kubectl CLI.
  • Manage cluster-level access control and configure PodSecurityPolicies.
  • Create services of type NodePort.
  • Run privileged pods

Prepare the Cluster for a Tanzu Kubernetes Cluster

Prior to create a TKC, you have to configure a Subscribed Content Library on the Supervisor Cluster. The virtual machine image that is used for the TKC nodes is pulled from this library.

  1. Open the vSphere Client and Navigate to Menu > Content Libraries and click the + Create button.
  2. Give the library a Name, eg. TKC
  3. Configure the Content Library as follows:
    Type: Subscribed content library
    Subscription URL: https://wp-content.vmware.com/v2/latest/lib.json
    Download content: immediately
  4. Press YES to confirm the The thumbprint of the certificate.
  5. Select the Datastore to be used for content library items.
  6. Press FINISH to end the wizard.
  7. The library is synced in the backgroud. Verify that you can see Images in the Content Library:
  8. Wait until the Download is finished.

    Note: If you have problems downloading the images, or the Sync Task fails, open a SSH connection to the vCenter and check /var/log/vmware/content-library/cls.log for errors.
  9. Navigate to Host and Clusters > Cluster > Configure > Namespaces > General and click ADD LIBRARY
  10. Select the Subscribed Contant Library and press OK

Provision a Tanzu Kubernetes Cluster

Create a new Namespace to be used for the TKC. Note that vSphere Pods and TKC can coexist in a namespace. You can also have multiple TKC in a single Namespace.

Open the vSphere Client and navigate to Workload Management > Namespaces and click NEW NAMESPACE.

Select the Kubernetes-enabled Cluster, give the Namespace a name and click CREATE.

Add a Storage Policy to the new Namespace.

Log in to the supervisor Cluster.

# kubectl vsphere login --server= -u administrator@vsphere.local

Logged in successfully.

You have access to the following contexts:

If the context you wish to use is not in this list, you may need to try
logging in again later, or contact your cluster administrator.

To change context, use `kubectl config use-context `

Switch to the tanzu Namespace/Context:

# kubectl config use-context tanzu 
Switched to context "tanzu".

Verify that you can see the Virtual Machine Images from the content library. Select and note the version you want to install. You can either use the full string (eg. v1.17.8+vmware.1-tkg.1.5417466), or a shortcut (eg. v1.17).

# kubectl get virtualmachineimages
NAME                                                         VERSION                           OSTYPE
ob-15957779-photon-3-k8s-v1.16.8---vmware.1-tkg.3.60d2ffd    v1.16.8+vmware.1-tkg.3.60d2ffd    vmwarePhoton64Guest
ob-16466772-photon-3-k8s-v1.17.7---vmware.1-tkg.1.154236c    v1.17.7+vmware.1-tkg.1.154236c    vmwarePhoton64Guest
ob-16545581-photon-3-k8s-v1.16.12---vmware.1-tkg.1.da7afe7   v1.16.12+vmware.1-tkg.1.da7afe7   vmwarePhoton64Guest
ob-16551547-photon-3-k8s-v1.17.8---vmware.1-tkg.1.5417466    v1.17.8+vmware.1-tkg.1.5417466    vmwarePhoton64Guest

Verify that you have access to a storageclass and note the name. When your vCenter configured Storage Policy Name has non-DNS compliant characters, they are replaced. The name as stated here (storagegold) is needed later.

# k get storageclasses
storagegold   csi.vsphere.vmware.com   Delete          Immediate           true                   6d22h

Choose size for Worker and Control Plane Nodes from the table.

guaranteed-xlarge432 GB16 GBYes
best-effort-xlarge432 GB16 GBNo
guaranteed-large416 GB16 GBYes
best-effort-large416 GB16 GBNo
guaranteed-medium28 GB16 GBYes
best-effort-medium28 GB16 GBNo
guaranteed-small24 GB16 GBYes
best-effort-small24 GB16 GBNo
guaranteed-xsmall22 GB16 GBYes
best-effort-xsmall22 GB16 GBNo

Use the information gathered above to create a YAML File.

apiVersion: run.tanzu.vmware.com/v1alpha1
kind: TanzuKubernetesCluster
  name: tkg-cluster-1
  namespace: tanzu
    version: v1.17
      count: 1
      class: best-effort-small
      storageClass: storagegold
      count: 3
      class: best-effort-small
      storageClass: storagegold
        name: calico
        cidrBlocks: [""]
        cidrBlocks: [""]
      classes: ["storagegold"]
      defaultClass: storagegold

Annotated YAML:

apiVersion: run.tanzu.vmware.com/v1alpha1
kind: TanzuKubernetesCluster
  name: tkg-cluster-1                      # Cluster Name
  namespace: tanzu                         # Namespace to deploy the Cluster
    version: v1.17                         # K8S version from available images
      count: 1                             # Number of Control Plane Nodes
      class: best-effort-small             # Control Plane Node size
      storageClass: storagegold            # Control Plane Node storage
      count: 3                             # Number of Worker Nodes
      class: best-effort-small             # Worker Nodes size
      storageClass: storagegold            # Worker Nodes Storage
        name: calico
        cidrBlocks: [""]       # Services Network CIDR (Must not overlap with the Supervisor Cluster!)
        cidrBlocks: [""]      # Pods CIDR (Must not overlap with the Supervisor Cluster!)
      classes: ["storagegold"]             # Storage Classes available in the TKC. Storage must be assigned to the Namespace
      defaultClass: storagegold            # Default Storage Class

Deploy the configuration file.

# kubectl apply -f tkg-cluster.yaml
tanzukubernetescluster.run.tanzu.vmware.com/tkg-cluster-1 created

The deployment will take some time. You can check the status with kubectl describe. Wait until all nodes are in the ready state. At this point, you can already note the API Endpoint IP address ( which will be used to connect to the Tanzu Kubernetes Cluster later.

# kubectl describe tkc tkg-cluster-1
Name:         tkg-cluster-1
Namespace:    tanzu
  Cluster API Status:
    API Endpoints:
      Port:  6443
    Phase:   provisioned
  Node Status:
    tkg-cluster-1-control-plane-6wv74:             ready
    tkg-cluster-1-workers-fl5nk-7ddcbdc677-4d7w9:  pending
    tkg-cluster-1-workers-fl5nk-7ddcbdc677-9kd26:  pending
    tkg-cluster-1-workers-fl5nk-7ddcbdc677-wqn45:  pending
  Phase:                                           running
  Vm Status:
    tkg-cluster-1-control-plane-6wv74:             ready
    tkg-cluster-1-workers-fl5nk-7ddcbdc677-4d7w9:  poweredon
    tkg-cluster-1-workers-fl5nk-7ddcbdc677-9kd26:  poweredon
    tkg-cluster-1-workers-fl5nk-7ddcbdc677-wqn45:  poweredon
# kubectl get tkc tkg-cluster-1
NAME            CONTROL PLANE   WORKER   DISTRIBUTION                     AGE     PHASE
tkg-cluster-1   1               3        v1.17.8+vmware.1-tkg.1.5417466   9m16s   running

During the deployment, head over to the vSphere Client. You can see a new icon for the TKC including Control Plane and Worker Node Virtual Machines.

Each Tanzu Kubernetes Cluster gets its own Tier-1 Gateway and Load Balancer instance in NSX-T.

When the deployment is finished, you can log in to the Tanzu Kubernetes Cluster using kubectl with the vSphere Plugin:

# kubectl vsphere login --server=SUPERVISOR-CLUSTER-CONTROL-PLANE-IP \
--tanzu-kubernetes-cluster-name TANZU-KUBERNETES-CLUSTER-NAME \
--tanzu-kubernetes-cluster-namespace SUPERVISOR-NAMESPACE-WHERE-THE-CLUSTER-IS-DEPLOYED \
--vsphere-username VCENTER-SSO-USER-NAME

Please note that for authentication with SSO, you have to use the Supervisor Cluster IP ( and not the TKC Cluster IP (

# kubectl vsphere login --server= --tanzu-kubernetes-cluster-name tkg-cluster-2 --tanzu-kubernetes-cluster-namespace tanzu --vsphere-username administrator@vsphere.local

Logged in successfully.

You have access to the following contexts:

If the context you wish to use is not in this list, you may need to try
logging in again later, or contact your cluster administrator.

When you view the context configuration, you can see that the tkg-cluster-2 context is using the TKC Cluster IP (

# kubectl config get-contexts
CURRENT   NAME              CLUSTER           AUTHINFO                                          NAMESPACE
          playground   wcp:   playground
          tanzu      wcp:   tanzu
*         tkg-cluster-2   wcp:

Now I want to deploy an Nginx Webserver. Create a new YAML file with the following content:

apiVersion: apps/v1
kind: Deployment
  name: webserver
  replicas: 2
      app: webserver
        app: webserver
      - image: nginx
        name: nginx
apiVersion: v1
kind: Service
    app: webserver
  name: webserver
  - port: 80
    protocol: TCP
    targetPort: 80
    app: webserver
  sessionAffinity: None
  type: LoadBalancer

The deployment will fail and no pods are created. Check the error message with kubectl get events:

# kubectl get events --sort-by=.metadata.creationTimestamp
LAST SEEN   TYPE      REASON                   OBJECT                            MESSAGE
7s          Warning   FailedCreate             replicaset/webserver-54cbbc6dd8   Error creating: pods "webserver-54cbbc6dd8-" is forbidden: unable to validate against any pod security policy: []

Error creating: pods "<NAME>" is forbidden: unable to validate against any pod security policy: []

The problem is that the SSO User used to log in to the TKC is not allowed to deploy privileged pods. To solve the issue, apply a ClusterRoleBinding that applies "vmware-system-privileged". Create a YAML with the following content:

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
  name: rolebinding-default-privileged-sa-ns_default
  namespace: default
  kind: ClusterRole
  name: psp:vmware-system-privileged
  apiGroup: rbac.authorization.k8s.io
- kind: Group
  apiGroup: rbac.authorization.k8s.io
  name: system:serviceaccounts

Apply the RoleBinding with kubectl.

# kubectl apply -f tanzu-rolebinding.yaml

Verify that Pods and the Service have been deployed successfully:

# kubectl get service
NAME         TYPE           CLUSTER-IP   EXTERNAL-IP       PORT(S)        AGE
kubernetes   ClusterIP    <none>            443/TCP        150m
supervisor   ClusterIP      None         <none>            6443/TCP       149m
webserver    LoadBalancer   80:32056/TCP   14s

# kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
webserver-d698d7bd6-4ljbw   1/1     Running   0          26s
webserver-d698d7bd6-cmqwp   1/1     Running   0          26s

You should be able to visit now.

vSphere with Kubernetes Guide

1 thought on “VMware vSphere with Kubernetes Guide Part 6 - Tanzu Kubernetes Cluster”

  1. Hi Florian,

    do follow your blog posts a long time and do like how and what you write.

    Did you came accross an issue of ImagePullBackOff STATUS during/after deploying a simple manifest?

    My worker node is obviously not able to access the docker repo. I don't know how to login in the worker node. Do you?

    According to William Lam's deployment script for vSphere with Tanzu using ALB, i did build this environment on top of two NSX-V segments (routable). One is used for management, the second for combined workload and frontend traffic. ALB was built and configured with the script. As long as i'm able to evaluate this, to me it's ok.


Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.