ESXTOP Guide

The esxtop utility provides a detailed look at how ESXi uses resources in real time. It is probably the most useful utility to troubleshoot performance issues on an ESXi host. The tool is available either on the ESXi host itself (#esxtop), or from remote systems like the vMA (#resxtop).

In vSphere 6.5, esxtop has 9 panes that can be enable with the following shotcuts:

  • c: CPU
  • m: Memory
  • v: Disk Virtual Machine
  • u: Disk Device
  • d: Disk Adapter
  • n: Network
  • i: Interrupt
  • p: Power Management
  • x: vSAN

CPU Panel

KeyMetricNameDescription
AIDIDThe ID of a running world. The world ID is typically cloaked and the group ID is displayed, unless the group is expanded with the "e" command. This also applies to groups with a single world. The ID is never identical to the GID.
BGIDGroup IDThe Group ID of a running world. A group is sometimes also referred as Resource Pool which has nothing in common with the Resource Pools you can configure in a DRS Cluster.
 
Press 'N' to sort by GID column.
CLWIDLeader World IDThe Leader World ID, also known as VMX Cartel ID for Virtual Machines. The LWID typcally the first world that has been started in a group.
DNAMENameThe name of a World's group. The number appended to the name is the LWID. When the group is expanded with the "e" command the name of the World is displayed.
ENWLDNumber of WorldsThe number of worlds running in a Group. NWLD is 1 when either the Group as a single process, or when the group has been expanded with the "e" command.
F%USEDUsedPercentage of physical CPU core cycles used by the world or group. %USED depends on the frequency the CPU core is running with and can be higher or lower compared to %RUN when the nominal (rated) frequency differs. Groups can also be higher than 100% when more vCPUs are configured or there is a high %SYS usage.
%USED = %RUN + %SYS - %OVRLP
 
Press 'U' to sort by %USED column.
F%RUNRunPercentage of total scheduled time for the world to run. %RUN does not account hyperthreading and system time. On a hyperthreading enabled server, %RUN can be twice as large as %USED.
The main difference between %RUN and %USED is that %RUN does not account system time.
F%SYSSystemPercentage of time spent by the ESXi VMkernel on behalf of the world to process interrupts and to perform other system activities. A high %SYS usually means high IO.
F%WAITWaitPercentage of time a world spent in the blocked or busy wait state. This percentage also includes %IDLE. The theoretical maximum of %WAIT is (NWLD*100). It's nothing wrong when there is a high value because the idle time is included in %WAIT.
100%= %WAIT + %RDY + %CSTP + %RUN
F%VMWAITVirtual Machine WaitThe total percentage of time a world spent in a blocked state waiting for events. The %VMWAIT metric is only available for vCPUS or combined in a VM group.
For vCPUS when a Virtual Machine is expandede with the "e" command, %WAIT = %VMWAIT + %IDLE + %SWPWT
F%RDYReadyThe percentage of time the world was ready to run but waiting in a queue to be scheduled. This may happen even when there is plenty of free CPU cycles when a VM CPU is administratively limited which is shown by %MLMTD.
To determine CPU contention from %RDY you have to take into account %RDY, %MLMTD and the number of vCPUS. If %RDY - %MLMTD is higher than 10% per vCPU you are very likely experiencing CPU contention. Typically you want to see %RDY close to 0.
100% = %RUN + %RDY + %CSTP + %WAIT
 
Press 'R' to sort by %RDY column.
F%IDLEIdleThe percentage of time the vCPU world is in an idle loop. %IDLE is only available for vCPU worlds. Other worlds do not have idle loops, so %IDLE is 0.
F%OVRLPOverlapThe percentage of time spent by system services on behalf of other worlds. For example, if VM A is currently being scheduled and a network packet for VM B is processed by the ESXi VMkernel, the time spent appears as %OVRLP for virtual machine A and %SYS for virtual machine B.
F%CSTPCoStopThe percentage of time that the VM is ready to run but is waiting for the availability of other vCPUs. The co-deschedule state applies only for SMP VMs. The CPU scheduler might put a vCPU in this state, when the VM workload does not use vCPUs in a balanced fashion. For example, if you have a VM with 2 vCPUs running a non SMP aware application, utilizing 1 vCPU at 100% and 1 vCPU at 0%. In that case, the CPU scheduler penalties the VM to reduce resource shortage for other VMs. This is is represented as %CSTP.
F%MLMTDMax LimitedThe percentage of time a world was ready to run but deliberately wasn't scheduled because doing so would violate the resource pool or virtual machine CPU limit. The %MLMTD time is included in %RDY time. If the value is not 0, the VM has been administratively limited.
F%SWPWTSwap WaitThe percentage of time a world spends waiting for the VMkernel to swap memory. The %SWPWT time is included in the %WAIT time. If %SWPWT is greater than 0, VMkernel is swapping the VMs memory to the disk. This will have a large negative impact in overall performance. Swapping can be caused by high memory oversubscription or memory limits configured on a resource pool or VM.
GSWTCH/sSwitches/secNumber of world switches per second. A context switch occurs when a CPU switches the execution from one world to another.
GMIG/sMigrates/secNumber of migrations per second. A world can be migrated from a busy pCPU to a less loaded pCPU for load balancing.
GQEXP/sQuantum Expires/secNumber of quantum expirations per second. This happens when a time quantum given to a currently running world expires. It is common that a world changes its state to WAIT before the current time quantum expires (50ms).
GWAKE/sWakeups/secNumber of world wakeups per second. A world wakes up when its state is changes from WAIT to READY
HAMINAlloc MinResource pool, virtual machine, or world attribute reservation. A value of 0 means no reservation.
HAMAXAlloc MaxResource pool, virtual machine, or world attribute limit. A value of -1 means unlimited.
HASHRSAlloc SharesResource pool, virtual machine, or world attribute Shares. Default shares are -4 (High), -3 (Normal) and -2 (Low). When a custom share is configured, the value is displayed.
HAMLMTAlloc Min LimitedUndocumented parameter.
HAUNITSAllocated unitAllocation unit (MHz for virtual machines)
I%LAT_CLatency CPUThe percentage of time the Resource Pool/World was ready to run but was not scheduled to run because of cpu resource contention.
I%LAT_MLatency MemoryThe percentage of time the Resource Pool/World was ready to run but was not scheduled to run because of memory resource contention.
I%DMDDemandThe CPU demand in percentage. It represents the average active CPU load in the past 1 minute.
IEMINEffective MinEffective minimum cpu allocation in case of resource contention. This is the amount of MHz the world will receieve when virtual machines are competing for resources.
ITIMER/sTimer/secTimer rate for this world.
IAFFINITY_BIT_MASKAffinity Bit MaskBit mask showing the current scheduling affinity for the world.
ICPUCPUThe physical or logical processor on which the world was running when resxtop (or esxtop) obtained this information.
IHTSHARINGHT SharingCurrent hyperthreading configuration.
IHTQHT QuarantineIndicates whether the world is currently quarantined or not. N means no and Y means yes.
JPOWERPower Usage WattsCurrent CPU power consumption for a virtual machine (in Watts). Usage calculation is based on the value provided by the power supply.

Memory Panel

KeyMetricNameDescription
AIDIDThe actual world ID is cloaked and the group ID is displayed. As you can't expand the group in the memory screen you can't see the real world ID.
BGIDGroup IDThe Group ID of a running world. A group is sometimes also referred as Resource Pool which has nothing in common with the Resource Pools you can configure in a DRS Cluster.
 
Press 'N' to sort by GID column.
CLWIDLeader World IDThe Leader World ID, also known as VMX Cartel ID for Virtual Machines. The LWID typcally the first world that has been started in a group.
DNAMENameThe name of a World's group. The number appended to the name is the LWID. Virtual Machines do not have a number appended.
ENWLDNumber of WorldsThe number of worlds running in the Group. NWLD is 1 when the Group as a single process.
FAMINAlloc MinResource pool, virtual machine, or world attribute reservation. A value of 0 means no reservation.
FAMAXAlloc MaxResource pool, virtual machine, or world attribute limit. A value of -1 means unlimited.
FASHRSAlloc SharesResource pool, virtual machine, or world attribute Shares. Default shares are -4 (High), -3 (Normal) and -2 (Low). When a custom share is configured, the value is displayed.
FAMLMTAlloc Min LimitedUndocumented parameter.
FAUNITSAllocated unitAllocation unit (kilobytes for virtual machines)
GNHNNuma Home NodesCurrent home node for the resource pool or virtual machine. This statistic is applicable only on NUMA systems. If the virtual machine has no home node, a dash (-) appears. A virtual machine runs only on processors within its home node, and its newly allocated memory comes from the home node as well. A virtual machine can have multiple home nodes when the number of virtual CPUs exceeds the number of cores per physical socket.
GNMIGNuma Rebalance Count Delta
GNRMEMNuma Remote Memory MBytesAmount of remote memory allocated to the virtual machine.
GNLMEMNuma Local Memory MBytesAmount of local memory allocated to the virtual machine.
GN%LNuma % LocalPercentage of local memory allocated to the virtual machine. This value should be close to 100%. A reason for poor NUMA locality can be that a virtual machine has more memory configured than it is available for each processor. Accessing remote memory causes increased latency.
HMEMSZMemory Size MBytesAmount of physical memory allocated to a virtual machine. This is the configured virtual machine memory.
MEMSZ = GRANT + MCTLSZ + SWCUR + untouched
 
Press 'M' to sort by MEMSZ column.
HGRANTMemory Granted Size MBytesAmount of physical memory mapped to the virtual machine. If GRANT is lower than MEMSZ, either the guest has never used all of its configured memory or if it has been reclaimed by the balloon driver.
HCNSMMemory Consumed Size MBytesAmount of the memory consumed by the virtual machine. The memory currently consumed by the virtual machine is equal to the amount of memory that the VM guest operating system currently uses, excluding the amount of memory saved by transparent page sharing and memory compression.
HSZTGTTarget Size MBytesTargeted amount of physical memory to be allocated. It is possible that SZTGT is higher than MEMSZ because it includes the virtual machine overhead memory.
HTCHDTouched MBytesThe amount of physical memory recently used (read or write) by the virtual machine. Touched memory is a working set estimate, which indicates how actively the VM is using its memory. This value is similar to the active memory reported by the guest OS.
HTCHD_WTouched Write MBytesThe amount of physical memory recently written by the virtual machine.
I%ACTVActive EstimatePercentage of active guest physical memory, current value.
I%ACTVSActive Slow EstimatePercentage of active guest physical memory, slow moving average.
I%ACTVFActive Fast EstimatePercentage of active guest physical memory, fast moving average.
I%ACTVNActive Next EstimatePercentage of active guest physical memory, predict of what %ACTVF will be at next sample.
JMCTL?Memctl?Memory balloon driver is installed or not. N means no, Y means yes. The ballooning driver is part of VMware Tools.
JMCTLSZMemctl MBytesAmount of physical memory reclaimed from the virtual machine by ballooning. To decrease host memory pressure, the ballon driver inflates inside the virtual machine and makes the physical memory available for other virtual machines. The performance impact caused by ballooning is small and therefore preferred over swapping. Another reason for ballooning can be a configured memory limit (AMAX).
 
Press 'B' to sort by MCTLSZ column.
JMCTLTGTMemctl Target MBytesAmount of physical memory the ESXi system attempts to reclaim from the virtual machine by way of ballooning.
JMCTLMAXMemctl Max MBytesThe maximum amount of guest physical memory reclaimable by the balloon driver.
KSWCURSwapped MBytesCurrent swap usage by this virtual machine.
KSWTGTSwap Target MBytesTarget where the ESXi host expects the swap usage by the virtual machine to be.
KSWR/sSwap Read MBytes/secRate at which the ESXi host swaps in memory from disk for the virtual machine.
KSWW/sSwap Written MBytes/secRate at which the ESXi host swaps virtual machine memory to disk.
LLLSWR/sLlswap Read MBytes/secThe rate at which memory is read from the host cache. When host caching is enabled, the ESXi host is able to swap to a local SSD, instead of the virtual machines datastore, which significantly reduces the impact caused by swapping.
LLLSWW/sLlswap Written MBytes/secRate at which memory is written to the host cache. When host caching is enabled, the ESXi host is able to swap to a local SSD, instead of the virtual machines datastore, which significantly reduces the impact caused by swapping.
MCPTRDCheckpoint Read MBytesThe amount of data read from the checkpoint file. Data from a checkpoint file is read when a virtual machine is resumed from a suspended state.
MCPTTGTCheckpoint Target MBytesThe size of checkpoint file. The checkpoint file is used when a virtual machine is suspended.
NZEROZero MBytesThe size of virtual machine physical pages that are zeroed. A zero page is simply a memory page that is all zeros which can be easily used for page sharing.
NSHRDShared MBytesAmount of guest physical memory that is shared.
NSHRDSVDShared Saved MBytesAmount of physical memory that is saved due to page sharing.
NCOWHCopy On Write Hint MBytesAmount of guest physical hint pages for page sharing.
OOVHDUWOverhead UW MBytesAmount of overhead memory reserved for the vmx user world.
OOVHDOverhead MBytesAmount of overhead memory currently consumed by the virtual machine.
OOVHDMAXOverhead Max MBytesAmount of reserved overhead memory for the entire VM.
PMCMTTGTMin Commit Target MBytes
PCMTTGTCommit Target MBytes
PCMTCHRGCommit Charged MBytes
PCMTPPSCommit Pages Per Share
QCACHESZCompressed Memory MBytesCurrent size of compressed memory by this virtual machine.
QCACHEUSDUsed Compressed Memory MBytesUsed compression memory cache.
QZIP/sCompression MBytes/secCompressed memory per second.
QUNZIP/sDecompression MBytes/secDecompressed memory per second.

Disk (Virtual Machine) Panel

KeyMetricNameDescription
AIDIDThe virtual SCSI Disk ID. The ID is typically cloaked and the group ID is displayed, unless the group is expanded with the "e" command. It is the same ID as used by vscsiStats.
BGIDGroup IDThe Group ID of the virtual machine.
CVMNAMEVirtual Machine NameThe display name of the virtual machine.
 
Press 'N' to sort by VMNAME column.
DVDEVNAMEVirtual Device NameName of the VSCSI device. Only displayed when the group is expanded with the "e" command.
ENVDISKNumber of Virtual DisksNumber of VSCSI devices.
FCMDS/sCommands/secNumber of commands issued per second.
FREADS/sReads/secNumber of read commands issued per second.
 
Press 'r' to sort by READS/s column.
FWRITES/sWrites/secNumber of write commands issued per second.
 
Press 'w' to sort by WRITES/s column.
FMBREAD/sMBytes Read/secMegabytes read per second.
 
Press 'R' to sort by MBREAD/s column.
FMBWRTN/sMBytes Written/secMegabytes written per second.
 
Press 'T' to sort by MBWRTN/s column.
GLAT/rdRead LatencyAverage latency (in milliseconds) per read.
HLAT/wrWrite LatencyAverage latency (in milliseconds) per write.

Disk (Device) Panel

KeyMetricNameDescription
ADEVICEDeviceName of the storage device.
BPATH/WORLD/PARTITIONPath/World/Partition IdThis column is only visible when expanded with either "e" for Disk World Statistics, "P" for Disk Path Statistics or "t" for Disk Partition Statistics.
CNPHNumber of PathsNumber of paths to the storage device.
CNWDNumber of WorldsNumber of worlds that are running on the storage device. Expand the device with the "e" command to display statistics for individual worlds.
CNPNNumber of PartitionsNumber of partitions on the storage device.
DSHARESSharesNumber of shares. Only displayed when the device is expanded with the "e" (Disk World Statistics) command.
EBLKSZBlock Size (Bytes)Physical Device block size in bytes.
ENUMBLKSNumber of BlocksNumber of blocks of the device.
FDQLENDevice Q DepthCurrent device queue depth of the storage device.
FWQLENWorld Q DepthWorld queue depth. This is the maximum number of ESXi VMkernel active commands that the world is allowed to have. This is a per device maximum for the world. It is valid only if the corresponding device is expanded to worlds.
FACTVActive CommandsNumber of commands in the ESXi VMkernel that are currently active. This statistic applies to only worlds and devices.
FQUEDQueued CommandsNumber of commands in the ESXi VMkernel that are currently queued. This statistic applies to only worlds and devices.
F%USD% UsedPercentage of the queue depth used by ESXi VMkernel active commands. This statistic applies to only worlds and devices.
FLOADLoadRatio of ESXi VMkernel active commands plus ESXi VMkernel queued commands to queue depth. This statistic applies to only worlds and devices.
GCMDS/sCommands/secNumber of commands issued per second.
GREADS/sReads/secNumber of read commands issued per second.
GWRITES/sWrites/secNumber of write commands issued per second.
GMBREAD/sMBytes Read/secMegabytes read per second.
GMBWRTN/sMBytes Written/secMegabytes written per second.
HRESV/sReserves/sec
HCONS/sConflicts/sec
IDAVG/cmdAverage Driver MilliSec/CommandThis is the average response time in milliseconds per command being sent to the device. The DAVG value is part of GAVG.
IKAVG/cmdAverage Kernel MilliSec/CommandThis is the amount of time the command spends in the VMkernel. The KAVG value is part of GAVG.
IGAVG/cmdAverage Guest MilliSec/Command"This is the response time as it is perceived by the guest operating system. This number is calculated with the formula: DAVG + KAVG = GAVG"
IQAVG/cmdAverage Queue MilliSec/CommandAverage queue latency per command in milliseconds.
JDAVG/rdAverage Driver MilliSec/ReadAverage device driver read latency per read operation in milliseconds.
JKAVG/rdAverage Kernel MilliSec/ReadAverage ESXi VMkernel read latency per read operation in milliseconds.
JGAVG/rdAverage Guest MilliSec/ReadAverage guest operating system read latency per read operation in milliseconds.
JQAVG/rdAverage Queue MilliSec/ReadAverage queue read latency per read operation in milliseconds.
KDAVG/wrAverage Driver MilliSec/WriteAverage device write latency per write operation in milliseconds.
KKAVG/wrAverage Kernel MilliSec/WriteAverage ESXi VMkernel write latency per write operation in milliseconds.
KGAVG/wrAverage Guest MilliSec/WriteAverage guest operating system write latency per write operation in milliseconds.
KQAVG/wrAverage Queue MilliSec/WriteAverage queue write latency per write operation in milliseconds.
LFCMDS/sFailed Commands/sec
LFREAD/sFailed Reads/sec
LFWRITE/sFailed Writes/sec
LFMBRD/sFailed Bytes Read/sec
LFMBWR/sFailed Bytes Written/sec
LFRESV/sFailed Reserves/sec
LABRTS/sAborts/secNumber of commands aborted per second.
LRESETS/sResets/secNumber of commands reset per second.
MPAECMD/sPAE Commands/secNumber of PAE commands per second. This statistic applies to only paths.
MPAECP/sPAE Copies/secNumber of PAE copies per second. This statistic applies to only paths.
NSPLTCMD/sSplit Commands/secNumber of split commands per second. This statistic applies to only paths.
NSPLTCP/sSplit Copies/secNumber of split copies per second. This statistic applies to only paths.
OCLONE_RDVAAI Clone Reads
OCLONE_WRVAAI Clone Writes
OCLONE_FVAAI Clone Failed
OMBC_RD/sMBytes Clone Reads/sec
OMBC_WR/sMBytes Clone Writes/sec
OATSAtomic Test and Set
OATSFAtomic Test and Set Failed
OZEROZeros
OZERO_FZeros Failed
OMBZERO/sMBytes Zeroed/sec
ODELETEDelete
ODELETE_FDelete Failed
OMBDEL/sMbytes Delete/sec
PCAVG/sucAverage Success Latency ms/Clone
PCAVG/fAverage Failure Latency ms/Clone
PAAVG/sucAverage Success Latency ms/ATS
PAAVG/fAverage Failure Latency ms/ATS
PZAVG/sucAverage Success Latency ms/Zero
PZAVG/fAverage Failure Latency ms/Zero

Disk (Adapter) Panel

KeyMetricNameDescription
AADAPTRAdapter
BPATHPath
CNPTHNumber of Paths
DAQLENAdapter Q Depth
ECMDS/sCommands/sec
EREADS/sReads/sec
EWRITES/sWrites/sec
EMBREAD/sMBytes Read/sec
EMBWRTN/sMBytes Written/sec
FRESV/sReserves/sec
FCONS/sConflicts/sec
GDAVG/cmdAverage Driver MilliSec/Command
GKAVG/cmdAverage Kernel MilliSec/Command
GGAVG/cmdAverage Guest MilliSec/Command
GQAVG/cmdAverage Queue MilliSec/Command
HDAVG/rdAverage Driver MilliSec/Read
HKAVG/rdAverage Kernel MilliSec/Read
HGAVG/rdAverage Guest MilliSec/Read
HQAVG/rdAverage Queue MilliSec/Read
IDAVG/wrAverage Driver MilliSec/Write
IKAVG/wrAverage Kernel MilliSec/Write
IGAVG/wrAverage Guest MilliSec/Write
IQAVG/wrAverage Queue MilliSec/Write
JFCMDS/sFailed Commands/sec
JFREAD/sFailed Reads/sec
JFWRITE/sFailed Writes/sec
JFMBRD/sFailed Bytes Read/sec
JFMBWR/sFailed Bytes Written/sec
JFRESV/sFailed Reserves/sec
JABRTS/sAborts/sec
JRESETS/sResets/sec
KPAECMD/sPAE Commands/sec
KPAECP/sPAE Copies/sec
LSPLTCMD/sSplit Commands/sec
LSPLTCP/sSplit Copies/sec

Network Panel

KeyMetricNameDescription
APORT-IDPort IDThe port ID used on the virtual switch. This ID is used for example for network traces with pktcap-uw.
 
Press 'N' to sort by PORT-ID column.
BUPLINKUplink?This virtual switch port is a physical uplink port. N means no, Y means yes.
CUPLink Up?Y means the corresponding link is up. N means it is not. Only visible for uplink ports.
CSPEEDLink Speed (Mb/s)Link speed in Megabits per second. Only visible for uplink ports.
CFDUPLXFull Duplex?Y means the corresponding link is operating at full duplex. N means it is not. Only visible for uplink ports.
DUSED-BYUsed ByIndicates what is connected to the virtual switch port. Connected devices can be virtual machine NICs, VMkernel ports (vmk#), physical NICs (vmnic#) or ports used for health checks (Shadow of vmnic#).
ETEAM-PNICTeam Uplink Physical NIC NameThe physical NIC that the corresponding device is actively using. This information is helpful for network troubleshooting.
FDNAMEDevice NameVirtual Switch name. Either the display name for virtual switches or DvsPortset-# for distributed switches.
GPKTTX/sPackets Transmitted/secNumber of packets transmitted per second.
 
Press 't' to sort by PKTTX/s column.
HMbTX/sMBits Transmitted/secMegaBits transmitted per second.
 
Press 'T' to sort by MbTX/s column.
IPSZTXAverage Packet Size Transmitted (Bytes)Average size of transmitted packets in bytes.
JPKTRX/sPackets Received/secNumber of packets received per second.
 
Press 'r' to sort by PKTRX/s column.
KMbRX/sMBits Received/secMegaBits received per second.
 
Press 'R' to sort by MbRX/s column.
LPSZRXAverage Packet Size Received (Bytes)Average size of received packets in bytes.
M%DRPTX% Outbound Packets DroppedPercentage of transmit packets dropped.
N%DRPRX% Received Packets DroppedPercentage of receive packets dropped.
OACTN/sActions Posted/secNumber of Vmkernel actions posted per second. This is an VMware internal counter with no further documentation.
PPKTTXMUL/sMulticast Packets Transmitted/secNumber of multicast packets transmitted per second.
PPKTRXMUL/sMulticast Packets Received/secNumber of multicast packets received per second.
QPKTTXBRD/sBroadcast Packets Transmitted/secNumber of broadcast packets transmitted per second.
QPKTRXBRD/sBroadcast Packets Received/secNumber of broadcast packets received per second.

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

This site uses Akismet to reduce spam. Learn how your comment data is processed.