ESXi 5 Network Troubleshooting Commands

Check if a remote host is online and reachable.

~ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=58 time=13.701 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=10.176 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=9.055 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 9.055/10.977/13.701 ms

Ping from a specific VMkernel adapter.

~ # ping -I vmk1 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=58 time=9.991 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=9.270 ms

Verify end-to-end MTU size. If you have jumbo frames configured in your environment, this might be useful. The -d option disables fragmentation, the -s option sets the packet size. Decrement the packet size until the ping succeeds. Add 28 Byte to the largest possible packet size (IP and ICMP headers). The result is your MTU. For Jumbo frames, the expected packet size is 8972 Bytes. In the following example the MTU is 1500 (Ping possible with 1472 Bytes +28 Bytes header).

~ # ping -d -s 1473 192.168.222.60
PING 192.168.222.60 (192.168.222.60): 1473 data bytes
sendto() failed (Message too long)

~ # ping -d -s 1472 192.168.222.60
PING 192.168.222.60 (192.168.222.60): 1472 data bytes
1480 bytes from 192.168.222.60: icmp_seq=0 ttl=64 time=0.885 ms
1480 bytes from 192.168.222.60: icmp_seq=1 ttl=64 time=0.913 ms

Display routing table

~ # /usr/sbin/esxcfg-route -l
VMkernel Routes:
Network          Netmask          Gateway          Interface
192.168.222.0    255.255.255.0    Local Subnet     vmk0
default          0.0.0.0          192.168.222.250  vmk0

Track the route packets taken to a given host. This is applicable for routed connections only.

~ # traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets
 1 c7200 (192.168.222.254) 0.716 ms 0.623 ms 0.592 ms
 2 hh-ea7-i.HH.DE.NET.DTAG.DE (62.154.32.70) 11.833 ms 11.297 ms 11.886 ms
 3 * 80.150.170.94 (80.150.170.94) 70.479 ms 70.370 ms
 4 72.14.233.130 (72.14.233.130) 8.755 ms 8.301 ms 8.612 ms
 5 google-public-dns-a.google.com (8.8.8.8) 7.652 ms 8.524 ms 8.343 ms

Display physical network adapters.

~ # esxcfg-nics -l
Name    PCI          Driver      Link Speed     Duplex MAC Address       MTU    Description
vmnic0  0000:07:00.0 tg3         Up   1000Mbps  Full   e4:11:5b:13:83:d3 1500   Broadcom Corporation NetXtreme BCM5723 Gigabit Ethernet
vmnic1  0000:03:00.0 igb         Down 0Mbps     Half   00:1b:21:93:b3:b0 1500   Intel Corporation 82576 Gigabit Network Connection
vmnic2  0000:03:00.1 igb         Down 0Mbps     Half   00:1b:21:93:b3:b1 1500   Intel Corporation 82576 Gigabit Network Connection
vmnic3  0000:04:00.0 igb         Down 0Mbps     Half   00:1b:21:93:b3:b2 1500   Intel Corporation 82576 Gigabit Network Connection
vmnic4  0000:04:00.1 igb         Up   1000Mbps  Full   00:1b:21:93:b3:b3 1500   Intel Corporation 82576 Gigabit Network Connection

Display physical network adapters including packet counters, ring parameters and driver information.

~ # /usr/lib/vmware/vm-support/bin/nicinfo.sh
NIC:  vmnic4

NICInfo:
   Advertised Auto Negotiation: true
   Advertised Link Modes: 10baseT/Half, 10baseT/Full, 100baseT/Half, 100baseT/Full, 1000baseT/Full
   Auto Negotiation: true
   Cable Type: Twisted Pair
   Current Message Level: 7
   Driver Info:
      NICDriverInfo:
         Bus Info: 0000:04:00.1
         Driver: igb
         Firmware Version: 1.2.1
         Version: 5.0.5.1
   Link Detected: true
   Link Status: Up
   Name: vmnic4
   PHY Address: 1
   Pause Autonegotiate: true
   Pause RX: false
   Pause TX: false
   Supported Ports: TP
   Supports Auto Negotiation: true
   Supports Pause: true
   Supports Wakeon: false
   Transceiver: internal
   Wakeon: None
Ring parameters for vmnic4:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             256
RX Mini:        0
RX Jumbo:       0
TX:             256


NIC statistics for vmnic4:
   Packets received: 31935
   Packets sent: 4499
   Bytes received: 3651845
   Bytes sent: 276356
   Receive packets dropped: 0
   Transmit packets dropped: 0
[...]

Display VMkernel adapters.

~ # esxcfg-vmknic -l
Interface  Port Group/DVPort/Opaque Network        IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type                NetStack
vmk0       Management Network                      IPv4      192.168.222.21                          255.255.255.0   192.168.222.255 e4:11:5b:13:83:d3 1500    65535     true    STATIC              defaultTcpipStack
vmk0       Management Network                      IPv6      fe80::e611:5bff:fe13:83d3               64                              e4:11:5b:13:83:d3 1500    65535     true    STATIC, PREFERRED   defaultTcpipStack
vmk1       VMkernel                                IPv4      192.168.222.27                          255.255.255.0   192.168.222.255 00:50:56:69:62:af 1500    65535     true    DHCP                defaultTcpipStack
vmk1       VMkernel                                IPv6      fe80::250:56ff:fe69:62af                64                              00:50:56:69:62:af 1500    65535     true    STATIC, PREFERRED   defaultTcpipStack

Display ARP table

~ # esxcli network ip neighbor list
Neighbor         Mac Address        Vmknic     Expiry  State  Type
---------------  -----------------  ------  ---------  -----  -------
192.168.222.50   bc:5f:f4:45:31:22  vmk0     1189 sec         Unknown
192.168.222.60   00:1b:21:93:b9:a4  vmk0      272 sec         Unknown
192.168.222.254  74:31:70:4e:d7:be  vmk0     1197 sec         Unknown
192.168.222.10   (incomplete)       vmk0       -1 sec         Unknown
fe80::1          74:31:70:4e:d7:be  vmk0    85926 sec  Stale  Unknown

Verify that the host can reach ports on external server (ESXi Port Scanner). Actually it is the netcat command
In this example I am verifying that the vCenters https port, and iSCSI from an external storage is accessible.

~ # nc -z 192.168.222.20 443
Connection to 192.168.222.20 443 port [tcp/https] succeeded!
~ # nc -z 192.168.222.60 3260
Connection to 192.168.222.60 3260 port [tcp/*] succeeded!

Collect packet traces from a specific VMkernel interface.

 tcpdump-uw -i vmk0

Collect packet traces on a specific protocol. This command displays ICMP (ping) only.

~ # tcpdump-uw icmp
tcpdump-uw: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmk0, link-type EN10MB (Ethernet), capture size 96 bytes
19:53:31.339259 IP truncated-ip - 2 bytes missing! 192.168.222.172 > esx1.virten.lab: ICMP echo request, id 237, seq 0, length 64
19:53:31.341207 IP truncated-ip - 2 bytes missing! esx1.virten.lab > 192.168.222.172: ICMP echo reply, id 237, seq 0, length 64
19:53:32.342857 IP truncated-ip - 2 bytes missing! 192.168.222.172 > esx1.virten.lab: ICMP echo request, id 237, seq 1, length 64
19:53:32.342918 IP truncated-ip - 2 bytes missing! esx1.virten.lab > 192.168.222.172: ICMP echo reply, id 237, seq 1, length 64
19:53:33.348021 IP truncated-ip - 2 bytes missing! 192.168.222.172 > esx1.virten.lab: ICMP echo request, id 237, seq 2, length 64
19:53:33.348103 IP truncated-ip - 2 bytes missing! esx1.virten.lab > 192.168.222.172: ICMP echo reply, id 237, seq 2, length 64

6 packets captured
6 packets received by filter
0 packets dropped by kernel

Write tcpdump packet traces to a file for later analysis.

~ # tcpdump-uw -w dump.cap

Display active TCP/UDP connections.

~ # esxcli network ip connection list
Proto  Recv Q  Send Q  Local Address                    Foreign Address       State        World ID  CC Algo  World Name
-----  ------  ------  -------------------------------  --------------------  -----------  --------  -------  ---------------
tcp         0       0  127.0.0.1:8307                   127.0.0.1:59448       ESTABLISHED     35309  newreno  hostd-worker
tcp         0     820  127.0.0.1:59448                  127.0.0.1:8307        ESTABLISHED     33932  newreno  rhttpproxy-work
tcp         0       0  127.0.0.1:443                    127.0.0.1:31031       ESTABLISHED     33934  newreno  rhttpproxy-work
tcp         0     795  127.0.0.1:31031                  127.0.0.1:443         ESTABLISHED    406071  newreno  python
tcp         0       0  192.168.222.21:80                192.168.222.50:51114  TIME_WAIT           0

Display virtual switch information

~ # /usr/sbin/esxcfg-vswitch -l
Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0         1536        8           128               1500    vmnic0,vmnic4

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  VMkernel              0        1           vmnic4,vmnic0
  VM Network            0        1           vmnic0,vmnic4
  Management Network    0        1           vmnic0,vmnic4

Verify SSL certificate information from remote hosts. This example checks the certificate from a vCenter Server.

~ # openssl s_client -connect 192.168.222.20:443
WARNING: can't open config file: /usr/ssl/openssl.cnf
CONNECTED(00000003)
depth=0 CN = vcsa6.virten.lab, C = US
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = vcsa6.virten.lab, C = US
verify error:num=27:certificate not trusted
verify return:1
depth=0 CN = vcsa6.virten.lab, C = US
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/CN=vcsa6.virten.lab/C=US
   i:/CN=CA, dc=vsphere,dc=local/C=US/O=vcsa6.virten.lab
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDWjCCAkKgAwIBAgIJAMAJLg/pmkZsMA0GCSqGSIb3DQEBCwUAMEoxIDAeBgNV
BAMMF0NBLCBkYz12c3BoZXJlLGRjPWxvY2FsMQswCQYDVQQGEwJVUzEZMBcGA1UE
CgwQdmNzYTYudmlydGVuLmxhYjAeFw0xNTAyMDQxNzQ5NDBaFw0yNTAxMjkxNzQ5
MzlaMCgxGTAXBgNVBAMMEHZjc2E2LnZpcnRlbi5sYWIxCzAJBgNVBAYTAlVTMIIB
IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnx1VKMwiaqBs3a2dhBWaeHXO
dkN3bfc1H55N2ykyzzo7KGhdIPMOiYeDvp8nk2lddYy/GgTWn6zmqAI2C2yldttr
ruwUTgw4Czc4nO5rp3eocJxEQiFygFZHROk6PrJwfXHWmJ2uVmBp0srAJikqZbSF
Nq43pO5YEmLYEexh/kGEkMN8163YjCS5snBFwZRlXruXlRBtOy6ohdMQQgRftUvH
WXpj/HOnri89/svU3AcNN290zMAoc8ONUy1Ab4XuCPu7evgkiSd+WZ2mwRtPsjOQ
DldVXw7WwYB1r5PmOH5Xct8MKDGOJIpoLjitBXyq/QIZmN0StBpbHnQMuExQ8QID
AQABo2UwYzAhBgNVHREEGjAYhwTAqN4UghB2Y3NhNi52aXJ0ZW4ubGFiMB8GA1Ud
IwQYMBaAFLxreDKShk2/xZ3QJXArId9lwLKdMB0GA1UdDgQWBBS2YYOWP3mKZs7l
SETXdxGRm2GCPDANBgkqhkiG9w0BAQsFAAOCAQEAmqig6LfZeBKena/pN/rlz31R
mswMab8bSAthNDIJFYc6vanzcesffYvObQ5j6wXCM+iWKLsB/r3PZAT9RvW90Uc7
T4XZjXTE7RWwcGBF7XLASIZegjaRdzZ8ZIgcd88UruFdJZCO8NvPA140EmCZQfkR
M31QHTFwJ8T+eWmbCYHmQrkfacPiomtLGaMj6EtLXxYi9PMY+ILTzyBv+nR26vai
OiWSWJJsa4QxzLW2LkUUoFSgFaza9pyk7O6qBj4dcE4Ihv9eMBXu/00DXhPewBy4
p9ubHbf6xyeqjNPDPV9Obu55k1Y79UhZ1nPAZ08uMQpwwC6ZpOuGFaPZTyBvkQ==
-----END CERTIFICATE-----
subject=/CN=vcsa6.virten.lab/C=US
issuer=/CN=CA, dc=vsphere,dc=local/C=US/O=vcsa6.virten.lab
---
No client certificate CA names sent
---
SSL handshake has read 996 bytes and written 623 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES256-GCM-SHA384
    Session-ID:
    Session-ID-ctx:
    Master-Key: B289EEF91BFF7572C9641A6735E1B2A8E750C9DAA7FE3DD9510FA4FCCC3D0FE200AFAB967C71E9370FE63EBA6012B5BF
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1423597583
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---
bad select 4
  1. Hi,

    there's a way to force the vmkping to go out a specific physical nic? For example: if I have a vDS with two vmnics, how can "vmkping/force" a vmkernel interface to use one of these 2 vmnics?

    Thanks

    • You can't force vmkping to a specific physical nic. You can only force it to use a vmkernel adapter (ping -I vmk1 8.8.8.8). A vmkernel adapter is bound to a physical interface and will (depending on the failover policy) always use that adapter. As a workaround, you could bind the dvPortgroup containing the vmkadapter to a specific dvUplink (containing your vmnic). (dvPortgroup Settings > Policies > Teaming and Failover). This will force your ping (and all other traffic!) to a specific physical interface.

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Trackbacks and Pingbacks: