vSphere with Kubernetes - Which Supervisor Cluster Settings can be edited?

When you want to deploy Kubernetes on vSphere 7 it is crucial to plan the configuration thoroughly prior to enabling Workload Management. Many of the configuration parameters entered during the Workload Management wizard can not be changed after the deployment.

The following table show which settings can be changed after the initial deployment:

Control Plane
Control Plane Size NO
Portgroup NO
IP Address NO
Subnet mask NO
Gateway NO
DNS Server YES
NTP Server YES
Search Domains NO
Workload Network
vSphere Distributed Switch NO
Edge Cluster NO
API Server endpoint FQDN NO
DNS Server YES
Pod CIDRs Add only
Service CIDRs NO
Ingress CIDRs Add only
Egress CIDRs Add only
Storage Policies
Control Plane Node YES
Ephemeral Disks YES
Image Cache YES

Control Plane Size

The Control Plane can be deployed in tiny, small, medium, and large. The control plane consists of 3 virtual machines that are deployed during the initial Workload Management configuration. The size does not only affect the virtual machine size but also the Kubernetes Pod and Service network size and maximum supported Services and Pods.

You can not change the Control Plane Size after the deployment.

The following table displays the configuration for each size.

Tiny Small Medium Large
Pod CIDR 10.244.0.0/21 10.244.0.0/20 10.244.0.0/19 10.244.0.0/18
Service CIDR 10.96.0.0/24 10.96.0.0/23 10.96.0.0/22 10.96.0.0/19
Controller CPU 2 4 8 16
Controller Memory 8 GB 16 GB 24 GB 32 GB
Controller Storage 16 GB 16 GB 16 GB 16 GB
Max Services 100 200 400 8000
Max Pods 1000 2000 4000 8000

Management Network

Step 3 in the Deployment Wizard, the Management Network configuration. These network settings are used to build the Controle Plane for the Supervisor Cluster.

The Management network, used for the Control Plane, can not be changed, except for DNS and NTP Servers.

To change DNS and NTP servers, open the vSphere Client and navigate to Hosts and Clusters > Cluster > Configure > Namespaces > Network and click EDIT next to Management Network Settings.

Workload Network

These network settings are used inside Kubernetes for Pods, Services, and to communicate with external networks.

You can't remove any configured CIDR ranges but you can add additional ranges for the following objects:

  • Pod CIDR
  • Ingress CIDR
  • Egress CIDR

You can not add Service networks, or change the Distributed Switch or Edge Cluster.

To add Pod/Ingress/Egress ranges, or change the DNS Server, open the vSphere Client and navigate to Hosts and Clusters > Cluster > Configure > Namespaces > Network and click EDIT next to Workload Network Settings.

As mentioned above, you can only add additional CIDR ranges. If you try to remove a previously configured range, the following error message is displayed. It does not matter if the network not in use:

When adding new ranges, make sure that there are no blanks in the list. At least until vCenter 7.0d, there is a UI bug. Wenn you open the configuration, it automatically adds a blank between networks. Just remove any blanks and the "Invalid Format: Expected single IP Address/Bits or a comma-separated list" is gone.

Storage Configuration

Step 4 in the Deployment Wizard, the Storage configuration for Control Plane Nodes, Ephemeral Disks, and Image Cache. Those settings can be changed at any time.

To change a Storage Policy, open the vSphere Client and navigate to Hosts and Clusters > Cluster > Configure > Namespaces > Storage and click EDIT next to Workload Network Settings.

If you change Storage Policies, keep in mind that it will not migrate running Containers or Control Plane Nodes. Only when deploying new pods, or restarting pods, they are compliant with the new Storage Policy. Storage vMotion does not work because Pods are configured with bus sharing. You should see the following message:

Virtual machine is configured to use a device that prevents the operation: Device 'SCSI controller 2' is a SCSI controller engaged in bus-sharing.

 

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

This site uses Akismet to reduce spam. Learn how your comment data is processed.