vSphere with Kubernetes - Which Supervisor Cluster Settings can be edited?

When you want to deploy Kubernetes on vSphere 7 it is crucial to plan the configuration thoroughly prior to enabling Workload Management. Many of the configuration parameters entered during the Workload Management wizard can not be changed after the deployment.

The following table show which settings can be changed after the initial deployment:

Control Plane Size

The Control Plane can be deployed in tiny, small, medium, and large. The control plane consists of 3 virtual machines that are deployed during the initial Workload Management configuration. The size does not only affect the virtual machine size but also the Kubernetes Pod and Service network size and maximum supported Services and Pods.

You can not change the Control Plane Size after the deployment.

The following table displays the configuration for each size.

Management Network

Step 3 in the Deployment Wizard, the Management Network configuration. These network settings are used to build the Controle Plane for the Supervisor Cluster.

The Management network, used for the Control Plane, can not be changed, except for DNS and NTP Servers.

To change DNS and NTP servers, open the vSphere Client and navigate to Hosts and Clusters > Cluster > Configure > Namespaces > Network and click EDIT next to Management Network Settings.

Workload Network

These network settings are used inside Kubernetes for Pods, Services, and to communicate with external networks.

You can't remove any configured CIDR ranges but you can add additional ranges for the following objects:

• Pod CIDR
• Ingress CIDR
• Egress CIDR

You can not add Service networks, or change the Distributed Switch or Edge Cluster.

To add Pod/Ingress/Egress ranges, or change the DNS Server, open the vSphere Client and navigate to Hosts and Clusters > Cluster > Configure > Namespaces > Network and click EDIT next to Workload Network Settings.

As mentioned above, you can only add additional CIDR ranges. If you try to remove a previously configured range, the following error message is displayed. It does not matter if the network not in use:

When adding new ranges, make sure that there are no blanks in the list. At least until vCenter 7.0d, there is a UI bug. Wenn you open the configuration, it automatically adds a blank between networks. Just remove any blanks and the "Invalid Format: Expected single IP Address/Bits or a comma-separated list" is gone.

Storage Configuration

Step 4 in the Deployment Wizard, the Storage configuration for Control Plane Nodes, Ephemeral Disks, and Image Cache. Those settings can be changed at any time.

To change a Storage Policy, open the vSphere Client and navigate to Hosts and Clusters > Cluster > Configure > Namespaces > Storage and click EDIT next to Workload Network Settings.

If you change Storage Policies, keep in mind that it will not migrate running Containers or Control Plane Nodes. Only when deploying new pods, or restarting pods, they are compliant with the new Storage Policy. Storage vMotion does not work because Pods are configured with bus sharing. You should see the following message:

Virtual machine is configured to use a device that prevents the operation: Device 'SCSI controller 2' is a SCSI controller engaged in bus-sharing.