VMware vSphere with Kubernetes Guide Part 4 - Working with kubectl

This is Part 4 of my "VMware vSphere with Kubernetes" Guide. In the last article, I've explained how to install and configure the Kubernetes CLI Tool kubectl and how to deploy the first pod. In this article, I'm taking a deeper look at kubectl.

This is a part of a series. If you do not have a Kubernetes activated vSphere Cluster, refer to Part 1 to get started with the deployment.


Configuration and Contexts

To recap from previous articles, I've used the following commands to login using the vSphere Plugin and worked with Namespaces. There is also a command to view the configuration which is saved in the .kube directory in your user's home directory.

## Login
# kubectl vsphere login --server=<CLUSTER> -u <USER>

## View a list of Contexts
# kubectl config get-contexts

## Switch to context
# kubectl config use-context playground

## View active context
# kubectl config current-context

## View Config
# kubectl config view

View Object Information

The 3-pod Nginx deployment from Part 3 is still running. Use kubectl get to view the state of all components.

# kubectl get deployments
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
webserver   3/3     3            3           2d

# kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
webserver-d698d7bd6-h495m   1/1     Running   0          2d
webserver-d698d7bd6-w86zd   1/1     Running   0          2d
webserver-d698d7bd6-wxlx6   1/1     Running   0          2d

# kubectl get services
NAME        TYPE           CLUSTER-IP   EXTERNAL-IP       PORT(S)        AGE
webserver   LoadBalancer   10.96.0.45   192.168.250.132   80:30733/TCP   2d

# kubectl get nodes
NAME                               STATUS   ROLES    AGE   VERSION
4217339ad6df885b6121ac551ba3db2d   Ready    master   9d    v1.17.4-2+a00aae1e6a4a69
4217931b9612c01232e50fe1eaa8cfb2   Ready    master   9d    v1.17.4-2+a00aae1e6a4a69
4217a39f121857906d9364ae52e9d91d   Ready    master   9d    v1.17.4-2+a00aae1e6a4a69
esx2.virten.lab                    Ready    agent    9d    v1.17.4-sph-c4c19c8
esx3.virten.lab                    Ready    agent    9d    v1.17.4-sph-c4c19c8
esx4.virten.lab                    Ready    agent    9d    v1.17.4-sph-c4c19c8
esx5.virten.lab                    Ready    agent    9d    v1.17.4-sph-c4c19c8

Many commands can be used with the option -o wide to get additional information.

# kubectl get pods  -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP             NODE              NOMINATED NODE   READINESS GATES
webserver-d698d7bd6-h495m   1/1     Running   0          2d    10.244.0.244   esx5.virten.lab   <none>           <none>
webserver-d698d7bd6-w86zd   1/1     Running   0          2d    10.244.0.242   esx5.virten.lab   <none>           <none>
webserver-d698d7bd6-wxlx6   1/1     Running   0          2d    10.244.0.243   esx5.virten.lab   <none>           <none>

You can also use the --all-namespaces option to list objects in all namespaces. There are about 60 pods just from enabling vSphere with Kubernetes.

# kubectl get pods --all-namespaces
NAMESPACE    NAME                                                              READY   STATUS    RESTARTS   AGE
kube-system  coredns-5cd65bfbdc-5sckk                          1/1     Running   0          9d
kube-system  coredns-5cd65bfbdc-gqs6b                          1/1     Running   0          9d
kube-system  coredns-5cd65bfbdc-prrq8                          1/1     Running   0          9d
kube-system  csr-signer-4217339ad6df885b6121ac551ba3db2d       1/1     Running   0          9d
kube-system  csr-signer-4217931b9612c01232e50fe1eaa8cfb2       1/1     Running   0          9d
kube-system  csr-signer-4217a39f121857906d9364ae52e9d91d       1/1     Running   0          9d
kube-system  docker-registry-4217339ad6df885b6121ac551ba3db2d  1/1     Running   0          9d
kube-system  docker-registry-4217931b9612c01232e50fe1eaa8cfb2  1/1     Running   0          9d
kube-system  docker-registry-4217a39f121857906d9364ae52e9d91d  1/1     Running   0          9d
[...]

To get the object's status and configuration in YAML, use the -o yaml option.

# kubectl get deployment webserver -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"webserver","namespace":"playground"},"spec":{"replicas":3,"selector":{"matchLabels":{"app":"webserver"}},"template":{"metadata":{"labels":{"app":"webserver"}},"spec":{"containers":[{"image":"nginx","name":"nginx"}]}}}}
  creationTimestamp: "2020-07-31T19:13:39Z"
  generation: 2
  name: webserver
  namespace: playground
  resourceVersion: "4222386"
  selfLink: /apis/apps/v1/namespaces/playground/deployments/webserver
  uid: bb3b8e09-4626-4802-a900-5352f914880b
spec:
  progressDeadlineSeconds: 600
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: webserver
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: webserver
    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: nginx
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 3
  conditions:
  - lastTransitionTime: "2020-07-31T19:13:39Z"
    lastUpdateTime: "2020-07-31T19:13:47Z"
    message: ReplicaSet "webserver-d698d7bd6" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2020-07-31T19:51:32Z"
    lastUpdateTime: "2020-07-31T19:51:32Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 2
  readyReplicas: 3
  replicas: 3
  updatedReplicas: 3

Describe Objects
The describe command can be used to get additional object information.

# kubectl describe deployment webserver
Name:                   webserver
Namespace:              playground
CreationTimestamp:      Fri, 31 Jul 2020 21:13:39 +0200
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=webserver
Replicas:               3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=webserver
  Containers:
   nginx:
    Image:        nginx
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   webserver-d698d7bd6 (3/3 replicas created)
Events:          <none>

# kubectl describe service webserver
Name:                     webserver
Namespace:                playground
Labels:                   app=webserver
Annotations:              ncp/internal_ip_for_policy: 100.64.0.1
Selector:                 app=webserver
Type:                     LoadBalancer
IP:                       10.96.0.45
LoadBalancer Ingress:     192.168.250.132
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  30733/TCP
Endpoints:                10.244.0.242:80,10.244.0.243:80,10.244.0.244:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Events and Logs

When you create a deployment, you do not see much output. To see what happens under the hood, use the kubectl get events command. This command is very important when pods can not be deployed as it will give hints where to start troubleshooting. Events are not displayed in chronological order, but they can be sorted with the --sort-by option.

#  kubectl get events --sort-by=.metadata.creationTimestamp
63s         Normal    SuccessfulRealizeNSXResource   pod/webserver-d698d7bd6-wxlx6                                 Successfully realized NSX resource for Pod
63s         Normal    SuccessfulRealizeNSXResource   pod/webserver-d698d7bd6-h495m                                 Successfully realized NSX resource for Pod
63s         Normal    SuccessfulRealizeNSXResource   pod/webserver-d698d7bd6-w86zd                                 Successfully realized NSX resource for Pod
40s         Normal    ScalingReplicaSet              deployment/webserver                                          Scaled up replica set webserver-d698d7bd6 to 3
40s         Normal    SuccessfulCreate               replicaset/webserver-d698d7bd6                                Created pod: webserver-d698d7bd6-9nlbx
<unknown>   Normal    Scheduled                      pod/webserver-d698d7bd6-jwjjm                                 Successfully assigned playground/webserver-d698d7bd6-jwjjm to esx5.virten.lab
40s         Normal    SuccessfulCreate               replicaset/webserver-d698d7bd6                                Created pod: webserver-d698d7bd6-jwjjm
40s         Normal    SuccessfulCreate               replicaset/webserver-d698d7bd6                                Created pod: webserver-d698d7bd6-csxkb
31s         Normal    SuccessfulRealizeNSXResource   pod/webserver-d698d7bd6-jwjjm                                 Successfully realized NSX resource for Pod
39s         Normal    Image                          pod/webserver-d698d7bd6-jwjjm                                 Image nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v49255 bound successfully
39s         Normal    Status                         image/nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v49255   esx3.virten.lab: Image status changed to Resolving
39s         Normal    Image                          pod/webserver-d698d7bd6-csxkb                                 Image nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v12196 bound successfully
31s         Normal    SuccessfulRealizeNSXResource   pod/webserver-d698d7bd6-9nlbx                                 Successfully realized NSX resource for Pod
<unknown>   Normal    Scheduled                      pod/webserver-d698d7bd6-9nlbx                                 Successfully assigned playground/webserver-d698d7bd6-9nlbx to esx5.virten.lab
39s         Normal    Image                          pod/webserver-d698d7bd6-9nlbx                                 Image nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v28985 bound successfully
<unknown>   Normal    Scheduled                      pod/webserver-d698d7bd6-csxkb                                 Successfully assigned playground/webserver-d698d7bd6-csxkb to esx5.virten.lab
31s         Normal    SuccessfulRealizeNSXResource   pod/webserver-d698d7bd6-csxkb                                 Successfully realized NSX resource for Pod
38s         Normal    Status                         image/nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v28985   esx3.virten.lab: Image status changed to Resolving
37s         Normal    Status                         image/nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v12196   esx2.virten.lab: Image status changed to Resolving
35s         Normal    Pulling                        pod/webserver-d698d7bd6-csxkb                                 Waiting for Image playground/nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v12196
35s         Normal    SuccessfulRealizeNSXResource   service/webserver                                             Successfully realized NSX resource for Service
35s         Normal    Pulling                        pod/webserver-d698d7bd6-9nlbx                                 Waiting for Image playground/nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v28985
35s         Normal    Pulling                        pod/webserver-d698d7bd6-jwjjm                                 Waiting for Image playground/nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v49255
34s         Normal    Bind                           image/nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v28985   Imagedisk f693fa68e7be60da5f2631f94268bb7231278a9e0a10e25798173ce6dd2e4d9d-v1487545 successfully bound
34s         Normal    Resolve                        image/nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v28985   esx3.virten.lab: Image resolved to ChainID sha256:f693fa68e7be60da5f2631f94268bb7231278a9e0a10e25798173ce6dd2e4d9d
34s         Normal    Status                         image/nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v28985   Image status changed to Ready
34s         Normal    Resolve                        image/nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v49255   esx3.virten.lab: Image resolved to ChainID sha256:f693fa68e7be60da5f2631f94268bb7231278a9e0a10e25798173ce6dd2e4d9d
34s         Normal    Bind                           image/nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v49255   Imagedisk f693fa68e7be60da5f2631f94268bb7231278a9e0a10e25798173ce6dd2e4d9d-v1487545 successfully bound
34s         Normal    Status                         image/nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v49255   Image status changed to Ready
34s         Normal    Pulled                         pod/webserver-d698d7bd6-9nlbx                                 Image playground/nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v28985 is ready
34s         Normal    Pulled                         pod/webserver-d698d7bd6-jwjjm                                 Image playground/nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v49255 is ready
33s         Normal    Bind                           image/nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v12196   Imagedisk f693fa68e7be60da5f2631f94268bb7231278a9e0a10e25798173ce6dd2e4d9d-v1487545 successfully bound
33s         Normal    Status                         image/nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v12196   Image status changed to Ready
33s         Normal    Resolve                        image/nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v12196   esx2.virten.lab: Image resolved to ChainID sha256:f693fa68e7be60da5f2631f94268bb7231278a9e0a10e25798173ce6dd2e4d9d
33s         Normal    Pulled                         pod/webserver-d698d7bd6-csxkb                                 Image playground/nginx-483640c55b8c28b65fdcc27be449ab8bf2454dd9-v12196 is ready
32s         Normal    Started                        pod/webserver-d698d7bd6-9nlbx                                 Started container nginx
32s         Normal    Started                        pod/webserver-d698d7bd6-jwjjm                                 Started container nginx
32s         Normal    Created                        pod/webserver-d698d7bd6-jwjjm                                 Created container nginx
32s         Normal    SuccessfulMountVolume          pod/webserver-d698d7bd6-9nlbx                                 Successfully mounted volume default-token-b7kc7
32s         Normal    SuccessfulMountVolume          pod/webserver-d698d7bd6-jwjjm                                 Successfully mounted volume default-token-b7kc7
32s         Normal    Created                        pod/webserver-d698d7bd6-9nlbx                                 Created container nginx
32s         Warning   FailedToUpdateEndpoint         endpoints/webserver                                           Failed to update endpoint playground/webserver: Operation cannot be fulfilled on endpoints "webserver": the object has been modified; please apply your changes to the latest version and try again
32s         Normal    Created                        pod/webserver-d698d7bd6-csxkb                                 Created container nginx
32s         Normal    Started                        pod/webserver-d698d7bd6-csxkb                                 Started container nginx
32s         Normal    SuccessfulMountVolume          pod/webserver-d698d7bd6-csxkb                                 Successfully mounted volume default-token-b7kc7

When pods are running you can use the kubectl logs <POD> command to get messages that have been written to stdout.

# kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
webserver-d698d7bd6-9nlbx   1/1     Running   0          18m
webserver-d698d7bd6-csxkb   1/1     Running   0          18m
webserver-d698d7bd6-jwjjm   1/1     Running   0          18m

# kubectl logs webserver-d698d7bd6-9nlbx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: error: ipv6 not available
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up

You can also stream logs similar to "tail -f" with the kubectl logs <POD> -f.

Interacting with Pods

Use kubectl exec to run commands inside containers.

# kubectl exec webserver-d698d7bd6-9nlbx -- ls -l /
total 68
drwxr-xr-x   2 root root 4096 Jul 20 00:00 bin
drwxr-xr-x   2 root root 4096 May  2 16:39 boot
drwxr-xr-x   1 root root  156 Aug  2 20:29 dev
drwxr-xr-x   1 root root 4096 Jul 22 03:23 docker-entrypoint.d
-rwxrwxr-x   1 root root 1202 Jul 22 03:22 docker-entrypoint.sh
drwxr-xr-x   1 root root   38 Aug  2 20:29 etc
drwxr-xr-x   2 root root 4096 May  2 16:39 home
drwxr-xr-x   1 root root 4096 Jul 22 03:23 lib
drwxr-xr-x   2 root root 4096 Jul 20 00:00 lib64
drwxr-xr-x   2 root root 4096 Jul 20 00:00 media
drwxr-xr-x   2 root root 4096 Jul 20 00:00 mnt
drwxr-xr-x   2 root root 4096 Jul 20 00:00 opt
dr-xr-xr-x 111 root root    0 Aug  2 20:29 proc
drwx------   1 root root   27 Aug  2 21:06 root
drwxr-xr-x   1 root root   38 Aug  2 20:29 run
drwxr-xr-x   2 root root 4096 Jul 20 00:00 sbin
drwxr-xr-x   2 root root 4096 Jul 20 00:00 srv
drwxr-xr-x   2 root root 4096 May  2 16:39 sys
drwxrwxrwt   1 root root 4096 Jul 22 03:23 tmp
drwxr-xr-x   1 root root 4096 Jul 20 00:00 usr
drwxr-xr-x   1 root root   19 Jul 20 00:00 var

The exec command can also be used to open an interactive shell inside the pod.

# kubectl exec --stdin --tty webserver-d698d7bd6-9nlbx  -- /bin/bash
root@webserver-d698d7bd6-9nlbx:/# ls -lh
total 68K
drwxr-xr-x   2 root root 4.0K Jul 20 00:00 bin
drwxr-xr-x   2 root root 4.0K May  2 16:39 boot
drwxr-xr-x   1 root root  156 Aug  2 20:29 dev
drwxr-xr-x   1 root root 4.0K Jul 22 03:23 docker-entrypoint.d
-rwxrwxr-x   1 root root 1.2K Jul 22 03:22 docker-entrypoint.sh
drwxr-xr-x   1 root root   38 Aug  2 20:29 etc
drwxr-xr-x   2 root root 4.0K May  2 16:39 home
drwxr-xr-x   1 root root 4.0K Jul 22 03:23 lib
drwxr-xr-x   2 root root 4.0K Jul 20 00:00 lib64
drwxr-xr-x   2 root root 4.0K Jul 20 00:00 media
drwxr-xr-x   2 root root 4.0K Jul 20 00:00 mnt
drwxr-xr-x   2 root root 4.0K Jul 20 00:00 opt
dr-xr-xr-x 113 root root    0 Aug  2 20:29 proc
drwx------   1 root root   27 Aug  2 21:06 root
drwxr-xr-x   1 root root   38 Aug  2 20:29 run
drwxr-xr-x   2 root root 4.0K Jul 20 00:00 sbin
drwxr-xr-x   2 root root 4.0K Jul 20 00:00 srv
drwxr-xr-x   2 root root 4.0K May  2 16:39 sys
drwxrwxrwt   1 root root 4.0K Jul 22 03:23 tmp
drwxr-xr-x   1 root root 4.0K Jul 20 00:00 usr
drwxr-xr-x   1 root root   19 Jul 20 00:00 var

 

vSphere with Kubernetes Guide

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

This site uses Akismet to reduce spam. Learn how your comment data is processed.