vSphere with Kubernetes - Which Supervisor Cluster Settings can be edited?

When you want to deploy Kubernetes on vSphere 7 it is crucial to plan the configuration thoroughly prior to enabling Workload Management. Many of the configuration parameters entered during the Workload Management wizard can not be changed after the deployment.

The following table show which settings can be changed after the initial deployment:

Control Plane
Control Plane SizeNO
PortgroupNO
IP AddressNO
Subnet maskNO
GatewayNO
DNS ServerYES
NTP ServerYES
Search DomainsNO
Workload Network
vSphere Distributed SwitchNO
Edge ClusterNO
API Server endpoint FQDNNO
DNS ServerYES
Pod CIDRsAdd only
Service CIDRsNO
Ingress CIDRsAdd only
Egress CIDRsAdd only
Storage Policies
Control Plane NodeYES
Ephemeral DisksYES
Image CacheYES

Control Plane Size

The Control Plane can be deployed in tiny, small, medium, and large. The control plane consists of 3 virtual machines that are deployed during the initial Workload Management configuration. The size does not only affect the virtual machine size but also the Kubernetes Pod and Service network size and maximum supported Services and Pods.

You can not change the Control Plane Size after the deployment.

The following table displays the configuration for each size.

TinySmallMediumLarge
Pod CIDR10.244.0.0/2110.244.0.0/2010.244.0.0/1910.244.0.0/18
Service CIDR10.96.0.0/2410.96.0.0/2310.96.0.0/2210.96.0.0/19
Controller CPU24816
Controller Memory8 GB16 GB24 GB32 GB
Controller Storage16 GB16 GB16 GB16 GB
Max Services1002004008000
Max Pods1000200040008000

Management Network

Step 3 in the Deployment Wizard, the Management Network configuration. These network settings are used to build the Controle Plane for the Supervisor Cluster.

The Management network, used for the Control Plane, can not be changed, except for DNS and NTP Servers.

To change DNS and NTP servers, open the vSphere Client and navigate to Hosts and Clusters > Cluster > Configure > Namespaces > Network and click EDIT next to Management Network Settings.

Workload Network

These network settings are used inside Kubernetes for Pods, Services, and to communicate with external networks.

You can't remove any configured CIDR ranges but you can add additional ranges for the following objects:

  • Pod CIDR
  • Ingress CIDR
  • Egress CIDR

You can not add Service networks, or change the Distributed Switch or Edge Cluster.

To add Pod/Ingress/Egress ranges, or change the DNS Server, open the vSphere Client and navigate to Hosts and Clusters > Cluster > Configure > Namespaces > Network and click EDIT next to Workload Network Settings.

As mentioned above, you can only add additional CIDR ranges. If you try to remove a previously configured range, the following error message is displayed. It does not matter if the network not in use:

When adding new ranges, make sure that there are no blanks in the list. At least until vCenter 7.0d, there is a UI bug. Wenn you open the configuration, it automatically adds a blank between networks. Just remove any blanks and the "Invalid Format: Expected single IP Address/Bits or a comma-separated list" is gone.

Storage Configuration

Step 4 in the Deployment Wizard, the Storage configuration for Control Plane Nodes, Ephemeral Disks, and Image Cache. Those settings can be changed at any time.

To change a Storage Policy, open the vSphere Client and navigate to Hosts and Clusters > Cluster > Configure > Namespaces > Storage and click EDIT next to Workload Network Settings.

If you change Storage Policies, keep in mind that it will not migrate running Containers or Control Plane Nodes. Only when deploying new pods, or restarting pods, they are compliant with the new Storage Policy. Storage vMotion does not work because Pods are configured with bus sharing. You should see the following message:

Virtual machine is configured to use a device that prevents the operation: Device 'SCSI controller 2' is a SCSI controller engaged in bus-sharing.

 

3 thoughts on “vSphere with Kubernetes - Which Supervisor Cluster Settings can be edited?”

  1. hello
    Hope you are doing fine. Just curious to know how Max pod and Max service count derived for each control plane VM size. VMware documentation seems to be silent on sizing. There seems to be some link between Pod CIDR and Service CIDR and these Max value of Pod and Service. Somehow i am not able to understand this link. Going through kubernetes guideline also to see if i can find something. Would be helpful if you can share some insight.

    1. There is a relationship when using default settings. You can get all default settings from the vCenter REST API: GET /api/vcenter/namespace-management/cluster-size-info

      If you don't want the mess with the API, you can easily call it from the vCenter UI: vCenter > Menu > Development Center > /api/vcenter/namespace-management/cluster-size-info > EXECUTE

      1. Thank you very much for taking time out for reply. I was able to get default setting information for pods and services

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.