vSphere 8 vCLS Machines High CPU Usage

Yesterday, vSphere 8.0 Update 1 has been released and as usual, before reading any release notes, I hit the update button. The update went well but after the update, I noticed that vCLS machines went rogue. One of the two machines had unusually high CPU usage.

There also was a stale vCLS machine which was neither powered on nor deleted.

To troubleshoot vCLS-related Problems, vSphere ESX Agent Manager (EAM) is your friend. The log, which is located at /var/log/vmware/eam/eam.log also had a weird Issue:

2023-04-20T13:22:26.354Z | ERROR | cluster-agent-3 | AuditedJob.java | 106 | JOB FAILED: [#1030641536] Upgrade ClusterAgentJob(ClusterAgent(ID: edaf955d-6b4f-4a72-8bd9-5831f6f7b888))
com.vmware.eam.job.DeployVmJob$DeployVmJobFailure: Can't provision VM for ClusterAgent(ID: edaf955d-6b4f-4a72-8bd9-5831f6f7b888) due to lack of suitable datastore.

Luckily, disabling and enabling vCLS for the cluster solved the issue immediately.

How to disable vCLS in vSphere 8

1. Get the Cluster ID, which looks like domain-c#. The easiest way to get it is by navigating to the Cluster using the vSphere Client and looking at the URL. (domain-c8 is this example)
   
2. To disable vCLS for a specific cluster, navigate to vCenter > Configure > Settings > Advanced Settings and press EDIT SETTINGS.
3. Add the following parameter (Replace with your Cluster ID)
   Name: config.vcls.clusters.domain-c8.enabled
   Value: False
4. Press ADD and SAVE
5. vCLS Machines should disappear after a couple of seconds.

To turn it back on, simply edit the configuration again and change the value to True. Do not add it again, search for it using the Name Filter and change the existing value.

You can also remove the entry, but this currently only works from the command line.

1. SSH to the vCSA and login as root
2. Edit /etc/vmware-vpx/vpxd.cfg (# vi /etc/vmware-vpx/vpxd.cfg)
3. Search the vcls secion (/vcls<ENTER>)
4. Remove the <vcls></vcls> section. Be careful to not mess up the file, remove exactly the following lines:
   
5. Save the file and exit (<ESC> :wq!)
6. Restart vpxd (# service-control --restart vmware-vpxd)

New vCLS Machines should be deployed shortly after removing the entry.

If this does not solve the issue, there might be a problem with SSL Trusts. See KB80469 for instructions on how to check and solve.