When you want to deploy Kubernetes on vSphere 7 it is crucial to plan the configuration thoroughly prior to enabling Workload Management. Many of the configuration parameters entered during the Workload Management wizard can not be changed after the deployment.
The following table show which settings can be changed after the initial deployment:
|Control Plane Size||NO|
|vSphere Distributed Switch||NO|
|API Server endpoint FQDN||NO|
|Pod CIDRs||Add only|
|Ingress CIDRs||Add only|
|Egress CIDRs||Add only|
|Control Plane Node||YES|
Control Plane Size
The Control Plane can be deployed in tiny, small, medium, and large. The control plane consists of 3 virtual machines that are deployed during the initial Workload Management configuration. The size does not only affect the virtual machine size but also the Kubernetes Pod and Service network size and maximum supported Services and Pods.
You can not change the Control Plane Size after the deployment.
The following table displays the configuration for each size.
|Controller Memory||8 GB||16 GB||24 GB||32 GB|
|Controller Storage||16 GB||16 GB||16 GB||16 GB|
Step 3 in the Deployment Wizard, the Management Network configuration. These network settings are used to build the Controle Plane for the Supervisor Cluster.
The Management network, used for the Control Plane, can not be changed, except for DNS and NTP Servers.
To change DNS and NTP servers, open the vSphere Client and navigate to Hosts and Clusters > Cluster > Configure > Namespaces > Network and click EDIT next to Management Network Settings.
These network settings are used inside Kubernetes for Pods, Services, and to communicate with external networks.
You can't remove any configured CIDR ranges but you can add additional ranges for the following objects:
- Pod CIDR
- Ingress CIDR
- Egress CIDR
You can not add Service networks, or change the Distributed Switch or Edge Cluster.
To add Pod/Ingress/Egress ranges, or change the DNS Server, open the vSphere Client and navigate to Hosts and Clusters > Cluster > Configure > Namespaces > Network and click EDIT next to Workload Network Settings.
As mentioned above, you can only add additional CIDR ranges. If you try to remove a previously configured range, the following error message is displayed. It does not matter if the network not in use:
When adding new ranges, make sure that there are no blanks in the list. At least until vCenter 7.0d, there is a UI bug. Wenn you open the configuration, it automatically adds a blank between networks. Just remove any blanks and the "Invalid Format: Expected single IP Address/Bits or a comma-separated list" is gone.
Step 4 in the Deployment Wizard, the Storage configuration for Control Plane Nodes, Ephemeral Disks, and Image Cache. Those settings can be changed at any time.
To change a Storage Policy, open the vSphere Client and navigate to Hosts and Clusters > Cluster > Configure > Namespaces > Storage and click EDIT next to Workload Network Settings.
If you change Storage Policies, keep in mind that it will not migrate running Containers or Control Plane Nodes. Only when deploying new pods, or restarting pods, they are compliant with the new Storage Policy. Storage vMotion does not work because Pods are configured with bus sharing. You should see the following message:
Virtual machine is configured to use a device that prevents the operation: Device 'SCSI controller 2' is a SCSI controller engaged in bus-sharing.
Hope you are doing fine. Just curious to know how Max pod and Max service count derived for each control plane VM size. VMware documentation seems to be silent on sizing. There seems to be some link between Pod CIDR and Service CIDR and these Max value of Pod and Service. Somehow i am not able to understand this link. Going through kubernetes guideline also to see if i can find something. Would be helpful if you can share some insight.
There is a relationship when using default settings. You can get all default settings from the vCenter REST API: GET
If you don't want the mess with the API, you can easily call it from the vCenter UI: vCenter > Menu > Development Center > /api/vcenter/namespace-management/cluster-size-info > EXECUTE
Thank you very much for taking time out for reply. I was able to get default setting information for pods and services