How to Migrate SupervisorControlPlaneVM in vSphere with Tanzu

When you try to migrate the Control Plane of a Workload Management enabled vSphere 7 cluster using vMotion or Storage vMotion, the following warning is displayed:

"This option is not available because you do not have the required permissions."

This article explains why manual migrations of the SupervisorControlPlaneVM shouldn't be necessary in general and how to work around the limitation if you still want to migrate it manually.

First of all, why shouldn't you bother migrating the Supervisor Control Plane (SCP)? The three Virtual Machines, that form the Control Plane of a vSphere with Tanzu enabled Kubernetes Cluster, are managed by the Workload Control Plane (WCP) which runs on the vCenter Server. This is a critical component of your Kubernetes deployment and must not be edited or touched at all from the vSphere Client.

• The WCP implementation keeps track of ESXi placement and will ensure anti-affinity.
• When an ESXi host running an SCP enters maintenance mode, it is automatically migrated.
• When an ESXi host running an SCP crashed, it is automatically restarted by VMware HA.
• The SCP is only responsible for and must run on the cluster it is managing. It can't be migrated to another Cluster.
• Storage Placement is controlled by policies. If you want to have the SCP on another datastore, you either have to remove the Datastore from the policy (usually by removing the tag), or by changing the default policy

Currently, the Storage Placement has an operational issue. Changing the default policy will only affect new objects. That means you have to trigger any type of redeployment. For the SCP, the only option at the moment is to perform an update as this will perform a rolling upgrade of all three VMs and they will be placed on a datastore, fitting to the new storage class.

To change the Storage Class, use the vSphere Client and navigate to Cluster > Configure > Namespaces > Storage. From here you can set the Storage Class for Control Plane Nodes (SCP VMs), Ephemeral Disks, and the Image Cache. But as mentioned, this will only affect new objects.

So, how can you migrate existing objects?

Note: The method described here is not supported. Use at your own risk! I recommend only using it in lab environments. As with the entire Tanzu implementation - Carefully plan your installation ahead!

Why can't the Global vCenter Administrator migrate VMs? The answer is simple, yet effective. They just overwrite the Administrator role by putting a more specific Read-only role on the Namespaces object with to Administrators group.

If you are now thinking that it would be an easy fix and can be fooled by creating a new Administrator that is not affected by the Read-only role, you will be surprised as the whole "Namespaces" Resource Pool and VM Folder will just vanish:

If you take a closer look at the SCP object permissions, you are going to find that several WCP users have appropriate permissions, so you just have to get their password.

How to get wcp-vmop-user password to manually migrate SCP VMs

1. SSH to the vCenter Server Appliance and login as root
2. Start the bash shell

   # shell

3. Get SupervisorControlPlane IP and credentials

   # /usr/lib/vmware-wcp/decryptK8Pwd.py
   
   Cluster: domain-c46:def22104-2b40-4048-b049-271b1de46b94
   IP: 10.99.2.10
   PWD: 3lnCN5ccPhg0cl1WQTZTGNzL[...]
   ------------------------------------------------------------
   

4. Use the IP and password from step 3. to login to the SupervisorControlPlane

   # ssh root@10.99.2.10

5. On the SupervisorControlPlane, you can run kubectl with kube-admin permissions. User Credentials are stored in secrets and the object we are looking for is wcp-vmop-sa-vc-auth in the vmware-system-vmop namespace.

   # kubectl get secrets wcp-vmop-sa-vc-auth -n vmware-system-vmop -o jsonpath='{.data}' |jq
   {
     "password": "XVBON2FvMyJSP0xpfidhckI/SGk=",
     "username": "d2NwLXZtb3AtdXNlci1kb21haW4tYzQ2LWY4YjE3Y2Y3LTM1NTQtNGFlZS1iNzIyLTg5NjQyZmMwNDk0ZkB2c3BoZXJlLmxvY2Fs"
   }
   

6. Username and Password are base64 encoded. You can simply decode them with "base64 -d"

   # kubectl get secrets wcp-vmop-sa-vc-auth -n vmware-system-vmop -o jsonpath='{.data.password}' |base64 -d
   ]PN7ao3"R?Li~'arB?Hi
   # kubectl get secrets wcp-vmop-sa-vc-auth -n vmware-system-vmop -o jsonpath='{.data.username}' |base64 -d
   wcp-vmop-user-domain-c46-f8b17cf7-3554-4aee-b722-89642fc0494f@vsphere.local
   

7. Use the WCP VM Operator to log in to your vCenter with the vSphere Client.
   
8. Now you can vMotion or Storage vMotion your Supervisor Control Plane VM.