ESXTOP Guide

The esxtop utility provides a detailed look at how ESXi uses resources in real time. It is probably the most useful utility to troubleshoot performance issues on an ESXi host. The tool is available either on the ESXi host itself (#esxtop), or from remote systems like the vMA (#resxtop).

In vSphere 6.5, esxtop has 9 panes that can be enable with the following shotcuts:

  • c: CPU
  • m: Memory
  • v: Disk Virtual Machine
  • u: Disk Device
  • d: Disk Adapter
  • n: Network
  • i: Interrupt
  • p: Power Management
  • x: vSAN

CPU Panel

Key Metric Name Description
A ID ID The ID of a running world. The world ID is typically cloaked and the group ID is displayed, unless the group is expanded with the "e" command. This also applies to groups with a single world. The ID is never identical to the GID.
B GID Group ID The Group ID of a running world. A group is sometimes also referred as Resource Pool which has nothing in common with the Resource Pools you can configure in a DRS Cluster.
 
Press 'N' to sort by GID column.
C LWID Leader World ID The Leader World ID, also known as VMX Cartel ID for Virtual Machines. The LWID typcally the first world that has been started in a group.
D NAME Name The name of a World's group. The number appended to the name is the LWID. When the group is expanded with the "e" command the name of the World is displayed.
E NWLD Number of Worlds The number of worlds running in a Group. NWLD is 1 when either the Group as a single process, or when the group has been expanded with the "e" command.
F %USED Used Percentage of physical CPU core cycles used by the world or group. %USED depends on the frequency the CPU core is running with and can be higher or lower compared to %RUN when the nominal (rated) frequency differs. Groups can also be higher than 100% when more vCPUs are configured or there is a high %SYS usage.
%USED = %RUN + %SYS - %OVRLP
 
Press 'U' to sort by %USED column.
F %RUN Run Percentage of total scheduled time for the world to run. %RUN does not account hyperthreading and system time. On a hyperthreading enabled server, %RUN can be twice as large as %USED.
The main difference between %RUN and %USED is that %RUN does not account system time.
F %SYS System Percentage of time spent by the ESXi VMkernel on behalf of the world to process interrupts and to perform other system activities. A high %SYS usually means high IO.
F %WAIT Wait Percentage of time a world spent in the blocked or busy wait state. This percentage also includes %IDLE. The theoretical maximum of %WAIT is (NWLD*100). It's nothing wrong when there is a high value because the idle time is included in %WAIT.
100%= %WAIT + %RDY + %CSTP + %RUN
F %VMWAIT Virtual Machine Wait The total percentage of time a world spent in a blocked state waiting for events. The %VMWAIT metric is only available for vCPUS or combined in a VM group.
For vCPUS when a Virtual Machine is expandede with the "e" command, %WAIT = %VMWAIT + %IDLE + %SWPWT
F %RDY Ready The percentage of time the world was ready to run but waiting in a queue to be scheduled. This may happen even when there is plenty of free CPU cycles when a VM CPU is administratively limited which is shown by %MLMTD.
To determine CPU contention from %RDY you have to take into account %RDY, %MLMTD and the number of vCPUS. If %RDY - %MLMTD is higher than 10% per vCPU you are very likely experiencing CPU contention. Typically you want to see %RDY close to 0.
100% = %RUN + %RDY + %CSTP + %WAIT
 
Press 'R' to sort by %RDY column.
F %IDLE Idle The percentage of time the vCPU world is in an idle loop. %IDLE is only available for vCPU worlds. Other worlds do not have idle loops, so %IDLE is 0.
F %OVRLP Overlap The percentage of time spent by system services on behalf of other worlds. For example, if VM A is currently being scheduled and a network packet for VM B is processed by the ESXi VMkernel, the time spent appears as %OVRLP for virtual machine A and %SYS for virtual machine B.
F %CSTP CoStop The percentage of time that the VM is ready to run but is waiting for the availability of other vCPUs. The co-deschedule state applies only for SMP VMs. The CPU scheduler might put a vCPU in this state, when the VM workload does not use vCPUs in a balanced fashion. For example, if you have a VM with 2 vCPUs running a non SMP aware application, utilizing 1 vCPU at 100% and 1 vCPU at 0%. In that case, the CPU scheduler penalties the VM to reduce resource shortage for other VMs. This is is represented as %CSTP.
F %MLMTD Max Limited The percentage of time a world was ready to run but deliberately wasn't scheduled because doing so would violate the resource pool or virtual machine CPU limit. The %MLMTD time is included in %RDY time. If the value is not 0, the VM has been administratively limited.
F %SWPWT Swap Wait The percentage of time a world spends waiting for the VMkernel to swap memory. The %SWPWT time is included in the %WAIT time. If %SWPWT is greater than 0, VMkernel is swapping the VMs memory to the disk. This will have a large negative impact in overall performance. Swapping can be caused by high memory oversubscription or memory limits configured on a resource pool or VM.
G SWTCH/s Switches/sec Number of world switches per second. A context switch occurs when a CPU switches the execution from one world to another.
G MIG/s Migrates/sec Number of migrations per second. A world can be migrated from a busy pCPU to a less loaded pCPU for load balancing.
G QEXP/s Quantum Expires/sec Number of quantum expirations per second. This happens when a time quantum given to a currently running world expires. It is common that a world changes its state to WAIT before the current time quantum expires (50ms).
G WAKE/s Wakeups/sec Number of world wakeups per second. A world wakes up when its state is changes from WAIT to READY
H AMIN Alloc Min Resource pool, virtual machine, or world attribute reservation. A value of 0 means no reservation.
H AMAX Alloc Max Resource pool, virtual machine, or world attribute limit. A value of -1 means unlimited.
H ASHRS Alloc Shares Resource pool, virtual machine, or world attribute Shares. Default shares are -4 (High), -3 (Normal) and -2 (Low). When a custom share is configured, the value is displayed.
H AMLMT Alloc Min Limited Undocumented parameter.
H AUNITS Allocated unit Allocation unit (MHz for virtual machines)
I %LAT_C Latency CPU The percentage of time the Resource Pool/World was ready to run but was not scheduled to run because of cpu resource contention.
I %LAT_M Latency Memory The percentage of time the Resource Pool/World was ready to run but was not scheduled to run because of memory resource contention.
I %DMD Demand The CPU demand in percentage. It represents the average active CPU load in the past 1 minute.
I EMIN Effective Min Effective minimum cpu allocation in case of resource contention. This is the amount of MHz the world will receieve when virtual machines are competing for resources.
I TIMER/s Timer/sec Timer rate for this world.
I AFFINITY_BIT_MASK Affinity Bit Mask Bit mask showing the current scheduling affinity for the world.
I CPU CPU The physical or logical processor on which the world was running when resxtop (or esxtop) obtained this information.
I HTSHARING HT Sharing Current hyperthreading configuration.
I HTQ HT Quarantine Indicates whether the world is currently quarantined or not. N means no and Y means yes.
J POWER Power Usage Watts Current CPU power consumption for a virtual machine (in Watts). Usage calculation is based on the value provided by the power supply.

Memory Panel

Key Metric Name Description
A ID ID The actual world ID is cloaked and the group ID is displayed. As you can't expand the group in the memory screen you can't see the real world ID.
B GID Group ID The Group ID of a running world. A group is sometimes also referred as Resource Pool which has nothing in common with the Resource Pools you can configure in a DRS Cluster.
 
Press 'N' to sort by GID column.
C LWID Leader World ID The Leader World ID, also known as VMX Cartel ID for Virtual Machines. The LWID typcally the first world that has been started in a group.
D NAME Name The name of a World's group. The number appended to the name is the LWID. Virtual Machines do not have a number appended.
E NWLD Number of Worlds The number of worlds running in the Group. NWLD is 1 when the Group as a single process.
F AMIN Alloc Min Resource pool, virtual machine, or world attribute reservation. A value of 0 means no reservation.
F AMAX Alloc Max Resource pool, virtual machine, or world attribute limit. A value of -1 means unlimited.
F ASHRS Alloc Shares Resource pool, virtual machine, or world attribute Shares. Default shares are -4 (High), -3 (Normal) and -2 (Low). When a custom share is configured, the value is displayed.
F AMLMT Alloc Min Limited Undocumented parameter.
F AUNITS Allocated unit Allocation unit (kilobytes for virtual machines)
G NHN Numa Home Nodes Current home node for the resource pool or virtual machine. This statistic is applicable only on NUMA systems. If the virtual machine has no home node, a dash (-) appears. A virtual machine runs only on processors within its home node, and its newly allocated memory comes from the home node as well. A virtual machine can have multiple home nodes when the number of virtual CPUs exceeds the number of cores per physical socket.
G NMIG Numa Rebalance Count Delta
G NRMEM Numa Remote Memory MBytes Amount of remote memory allocated to the virtual machine.
G NLMEM Numa Local Memory MBytes Amount of local memory allocated to the virtual machine.
G N%L Numa % Local Percentage of local memory allocated to the virtual machine. This value should be close to 100%. A reason for poor NUMA locality can be that a virtual machine has more memory configured than it is available for each processor. Accessing remote memory causes increased latency.
H MEMSZ Memory Size MBytes Amount of physical memory allocated to a virtual machine. This is the configured virtual machine memory.
MEMSZ = GRANT + MCTLSZ + SWCUR + untouched
 
Press 'M' to sort by MEMSZ column.
H GRANT Memory Granted Size MBytes Amount of physical memory mapped to the virtual machine. If GRANT is lower than MEMSZ, either the guest has never used all of its configured memory or if it has been reclaimed by the balloon driver.
H CNSM Memory Consumed Size MBytes Amount of the memory consumed by the virtual machine. The memory currently consumed by the virtual machine is equal to the amount of memory that the VM guest operating system currently uses, excluding the amount of memory saved by transparent page sharing and memory compression.
H SZTGT Target Size MBytes Targeted amount of physical memory to be allocated. It is possible that SZTGT is higher than MEMSZ because it includes the virtual machine overhead memory.
H TCHD Touched MBytes The amount of physical memory recently used (read or write) by the virtual machine. Touched memory is a working set estimate, which indicates how actively the VM is using its memory. This value is similar to the active memory reported by the guest OS.
H TCHD_W Touched Write MBytes The amount of physical memory recently written by the virtual machine.
I %ACTV Active Estimate Percentage of active guest physical memory, current value.
I %ACTVS Active Slow Estimate Percentage of active guest physical memory, slow moving average.
I %ACTVF Active Fast Estimate Percentage of active guest physical memory, fast moving average.
I %ACTVN Active Next Estimate Percentage of active guest physical memory, predict of what %ACTVF will be at next sample.
J MCTL? Memctl? Memory balloon driver is installed or not. N means no, Y means yes. The ballooning driver is part of VMware Tools.
J MCTLSZ Memctl MBytes Amount of physical memory reclaimed from the virtual machine by ballooning. To decrease host memory pressure, the ballon driver inflates inside the virtual machine and makes the physical memory available for other virtual machines. The performance impact caused by ballooning is small and therefore preferred over swapping. Another reason for ballooning can be a configured memory limit (AMAX).
 
Press 'B' to sort by MCTLSZ column.
J MCTLTGT Memctl Target MBytes Amount of physical memory the ESXi system attempts to reclaim from the virtual machine by way of ballooning.
J MCTLMAX Memctl Max MBytes The maximum amount of guest physical memory reclaimable by the balloon driver.
K SWCUR Swapped MBytes Current swap usage by this virtual machine.
K SWTGT Swap Target MBytes Target where the ESXi host expects the swap usage by the virtual machine to be.
K SWR/s Swap Read MBytes/sec Rate at which the ESXi host swaps in memory from disk for the virtual machine.
K SWW/s Swap Written MBytes/sec Rate at which the ESXi host swaps virtual machine memory to disk.
L LLSWR/s Llswap Read MBytes/sec The rate at which memory is read from the host cache. When host caching is enabled, the ESXi host is able to swap to a local SSD, instead of the virtual machines datastore, which significantly reduces the impact caused by swapping.
L LLSWW/s Llswap Written MBytes/sec Rate at which memory is written to the host cache. When host caching is enabled, the ESXi host is able to swap to a local SSD, instead of the virtual machines datastore, which significantly reduces the impact caused by swapping.
M CPTRD Checkpoint Read MBytes The amount of data read from the checkpoint file. Data from a checkpoint file is read when a virtual machine is resumed from a suspended state.
M CPTTGT Checkpoint Target MBytes The size of checkpoint file. The checkpoint file is used when a virtual machine is suspended.
N ZERO Zero MBytes The size of virtual machine physical pages that are zeroed. A zero page is simply a memory page that is all zeros which can be easily used for page sharing.
N SHRD Shared MBytes Amount of guest physical memory that is shared.
N SHRDSVD Shared Saved MBytes Amount of physical memory that is saved due to page sharing.
N COWH Copy On Write Hint MBytes Amount of guest physical hint pages for page sharing.
O OVHDUW Overhead UW MBytes Amount of overhead memory reserved for the vmx user world.
O OVHD Overhead MBytes Amount of overhead memory currently consumed by the virtual machine.
O OVHDMAX Overhead Max MBytes Amount of reserved overhead memory for the entire VM.
P MCMTTGT Min Commit Target MBytes
P CMTTGT Commit Target MBytes
P CMTCHRG Commit Charged MBytes
P CMTPPS Commit Pages Per Share
Q CACHESZ Compressed Memory MBytes Current size of compressed memory by this virtual machine.
Q CACHEUSD Used Compressed Memory MBytes Used compression memory cache.
Q ZIP/s Compression MBytes/sec Compressed memory per second.
Q UNZIP/s Decompression MBytes/sec Decompressed memory per second.

Disk (Virtual Machine) Panel

Key Metric Name Description
A ID ID The virtual SCSI Disk ID. The ID is typically cloaked and the group ID is displayed, unless the group is expanded with the "e" command. It is the same ID as used by vscsiStats.
B GID Group ID The Group ID of the virtual machine.
C VMNAME Virtual Machine Name The display name of the virtual machine.
 
Press 'N' to sort by VMNAME column.
D VDEVNAME Virtual Device Name Name of the VSCSI device. Only displayed when the group is expanded with the "e" command.
E NVDISK Number of Virtual Disks Number of VSCSI devices.
F CMDS/s Commands/sec Number of commands issued per second.
F READS/s Reads/sec Number of read commands issued per second.
 
Press 'r' to sort by READS/s column.
F WRITES/s Writes/sec Number of write commands issued per second.
 
Press 'w' to sort by WRITES/s column.
F MBREAD/s MBytes Read/sec Megabytes read per second.
 
Press 'R' to sort by MBREAD/s column.
F MBWRTN/s MBytes Written/sec Megabytes written per second.
 
Press 'T' to sort by MBWRTN/s column.
G LAT/rd Read Latency Average latency (in milliseconds) per read.
H LAT/wr Write Latency Average latency (in milliseconds) per write.

Disk (Device) Panel

Key Metric Name Description
A DEVICE Device Name of the storage device.
B PATH/WORLD/PARTITION Path/World/Partition Id This column is only visible when expanded with either "e" for Disk World Statistics, "P" for Disk Path Statistics or "t" for Disk Partition Statistics.
C NPH Number of Paths Number of paths to the storage device.
C NWD Number of Worlds Number of worlds that are running on the storage device. Expand the device with the "e" command to display statistics for individual worlds.
C NPN Number of Partitions Number of partitions on the storage device.
D SHARES Shares Number of shares. Only displayed when the device is expanded with the "e" (Disk World Statistics) command.
E BLKSZ Block Size (Bytes) Physical Device block size in bytes.
E NUMBLKS Number of Blocks Number of blocks of the device.
F DQLEN Device Q Depth Current device queue depth of the storage device.
F WQLEN World Q Depth World queue depth. This is the maximum number of ESXi VMkernel active commands that the world is allowed to have. This is a per device maximum for the world. It is valid only if the corresponding device is expanded to worlds.
F ACTV Active Commands Number of commands in the ESXi VMkernel that are currently active. This statistic applies to only worlds and devices.
F QUED Queued Commands Number of commands in the ESXi VMkernel that are currently queued. This statistic applies to only worlds and devices.
F %USD % Used Percentage of the queue depth used by ESXi VMkernel active commands. This statistic applies to only worlds and devices.
F LOAD Load Ratio of ESXi VMkernel active commands plus ESXi VMkernel queued commands to queue depth. This statistic applies to only worlds and devices.
G CMDS/s Commands/sec Number of commands issued per second.
G READS/s Reads/sec Number of read commands issued per second.
G WRITES/s Writes/sec Number of write commands issued per second.
G MBREAD/s MBytes Read/sec Megabytes read per second.
G MBWRTN/s MBytes Written/sec Megabytes written per second.
H RESV/s Reserves/sec
H CONS/s Conflicts/sec
I DAVG/cmd Average Driver MilliSec/Command This is the average response time in milliseconds per command being sent to the device. The DAVG value is part of GAVG.
I KAVG/cmd Average Kernel MilliSec/Command This is the amount of time the command spends in the VMkernel. The KAVG value is part of GAVG.
I GAVG/cmd Average Guest MilliSec/Command "This is the response time as it is perceived by the guest operating system. This number is calculated with the formula: DAVG + KAVG = GAVG"
I QAVG/cmd Average Queue MilliSec/Command Average queue latency per command in milliseconds.
J DAVG/rd Average Driver MilliSec/Read Average device driver read latency per read operation in milliseconds.
J KAVG/rd Average Kernel MilliSec/Read Average ESXi VMkernel read latency per read operation in milliseconds.
J GAVG/rd Average Guest MilliSec/Read Average guest operating system read latency per read operation in milliseconds.
J QAVG/rd Average Queue MilliSec/Read Average queue read latency per read operation in milliseconds.
K DAVG/wr Average Driver MilliSec/Write Average device write latency per write operation in milliseconds.
K KAVG/wr Average Kernel MilliSec/Write Average ESXi VMkernel write latency per write operation in milliseconds.
K GAVG/wr Average Guest MilliSec/Write Average guest operating system write latency per write operation in milliseconds.
K QAVG/wr Average Queue MilliSec/Write Average queue write latency per write operation in milliseconds.
L FCMDS/s Failed Commands/sec
L FREAD/s Failed Reads/sec
L FWRITE/s Failed Writes/sec
L FMBRD/s Failed Bytes Read/sec
L FMBWR/s Failed Bytes Written/sec
L FRESV/s Failed Reserves/sec
L ABRTS/s Aborts/sec Number of commands aborted per second.
L RESETS/s Resets/sec Number of commands reset per second.
M PAECMD/s PAE Commands/sec Number of PAE commands per second. This statistic applies to only paths.
M PAECP/s PAE Copies/sec Number of PAE copies per second. This statistic applies to only paths.
N SPLTCMD/s Split Commands/sec Number of split commands per second. This statistic applies to only paths.
N SPLTCP/s Split Copies/sec Number of split copies per second. This statistic applies to only paths.
O CLONE_RD VAAI Clone Reads
O CLONE_WR VAAI Clone Writes
O CLONE_F VAAI Clone Failed
O MBC_RD/s MBytes Clone Reads/sec
O MBC_WR/s MBytes Clone Writes/sec
O ATS Atomic Test and Set
O ATSF Atomic Test and Set Failed
O ZERO Zeros
O ZERO_F Zeros Failed
O MBZERO/s MBytes Zeroed/sec
O DELETE Delete
O DELETE_F Delete Failed
O MBDEL/s Mbytes Delete/sec
P CAVG/suc Average Success Latency ms/Clone
P CAVG/f Average Failure Latency ms/Clone
P AAVG/suc Average Success Latency ms/ATS
P AAVG/f Average Failure Latency ms/ATS
P ZAVG/suc Average Success Latency ms/Zero
P ZAVG/f Average Failure Latency ms/Zero

Disk (Adapter) Panel

Key Metric Name Description
A ADAPTR Adapter
B PATH Path
C NPTH Number of Paths
D AQLEN Adapter Q Depth
E CMDS/s Commands/sec
E READS/s Reads/sec
E WRITES/s Writes/sec
E MBREAD/s MBytes Read/sec
E MBWRTN/s MBytes Written/sec
F RESV/s Reserves/sec
F CONS/s Conflicts/sec
G DAVG/cmd Average Driver MilliSec/Command
G KAVG/cmd Average Kernel MilliSec/Command
G GAVG/cmd Average Guest MilliSec/Command
G QAVG/cmd Average Queue MilliSec/Command
H DAVG/rd Average Driver MilliSec/Read
H KAVG/rd Average Kernel MilliSec/Read
H GAVG/rd Average Guest MilliSec/Read
H QAVG/rd Average Queue MilliSec/Read
I DAVG/wr Average Driver MilliSec/Write
I KAVG/wr Average Kernel MilliSec/Write
I GAVG/wr Average Guest MilliSec/Write
I QAVG/wr Average Queue MilliSec/Write
J FCMDS/s Failed Commands/sec
J FREAD/s Failed Reads/sec
J FWRITE/s Failed Writes/sec
J FMBRD/s Failed Bytes Read/sec
J FMBWR/s Failed Bytes Written/sec
J FRESV/s Failed Reserves/sec
J ABRTS/s Aborts/sec
J RESETS/s Resets/sec
K PAECMD/s PAE Commands/sec
K PAECP/s PAE Copies/sec
L SPLTCMD/s Split Commands/sec
L SPLTCP/s Split Copies/sec

Network Panel

Key Metric Name Description
A PORT-ID Port ID The port ID used on the virtual switch. This ID is used for example for network traces with pktcap-uw.
 
Press 'N' to sort by PORT-ID column.
B UPLINK Uplink? This virtual switch port is a physical uplink port. N means no, Y means yes.
C UP Link Up? Y means the corresponding link is up. N means it is not. Only visible for uplink ports.
C SPEED Link Speed (Mb/s) Link speed in Megabits per second. Only visible for uplink ports.
C FDUPLX Full Duplex? Y means the corresponding link is operating at full duplex. N means it is not. Only visible for uplink ports.
D USED-BY Used By Indicates what is connected to the virtual switch port. Connected devices can be virtual machine NICs, VMkernel ports (vmk#), physical NICs (vmnic#) or ports used for health checks (Shadow of vmnic#).
E TEAM-PNIC Team Uplink Physical NIC Name The physical NIC that the corresponding device is actively using. This information is helpful for network troubleshooting.
F DNAME Device Name Virtual Switch name. Either the display name for virtual switches or DvsPortset-# for distributed switches.
G PKTTX/s Packets Transmitted/sec Number of packets transmitted per second.
 
Press 't' to sort by PKTTX/s column.
H MbTX/s MBits Transmitted/sec MegaBits transmitted per second.
 
Press 'T' to sort by MbTX/s column.
I PSZTX Average Packet Size Transmitted (Bytes) Average size of transmitted packets in bytes.
J PKTRX/s Packets Received/sec Number of packets received per second.
 
Press 'r' to sort by PKTRX/s column.
K MbRX/s MBits Received/sec MegaBits received per second.
 
Press 'R' to sort by MbRX/s column.
L PSZRX Average Packet Size Received (Bytes) Average size of received packets in bytes.
M %DRPTX % Outbound Packets Dropped Percentage of transmit packets dropped.
N %DRPRX % Received Packets Dropped Percentage of receive packets dropped.
O ACTN/s Actions Posted/sec Number of Vmkernel actions posted per second. This is an VMware internal counter with no further documentation.
P PKTTXMUL/s Multicast Packets Transmitted/sec Number of multicast packets transmitted per second.
P PKTRXMUL/s Multicast Packets Received/sec Number of multicast packets received per second.
Q PKTTXBRD/s Broadcast Packets Transmitted/sec Number of broadcast packets transmitted per second.
Q PKTRXBRD/s Broadcast Packets Received/sec Number of broadcast packets received per second.

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>