ESXTOP(1)

NAME
esxtop, resxtop - display ESXi resource utilization statistics

SYNOPSIS
esxtop [-] [h] [v] [b] [l] [s] [a] [c configfile] [R vm-support_dir_path] [d delay] [n iter] resxtop [-] [h] [v] [b] [s] [a] [c configfile] [d delay] [n iter] [-server server-name [-vihost host-name]] [-portnumber socket-port] [-username user-name]

DESCRIPTION
esxtop allows monitoring and collection of data for all system resources: CPU, memory, disk, network, etc. When used interactively, this data can be viewed on different types of screens; one each for CPU statistics, memory statistics, network statistics, disk adapter statistics, disk device statistics, disk VM statistics, interrupt statistics, power statistics and vSAN (VSAN) statistics. In the batch mode, data can be redirected to a file for offline uses. Many esxtop statistics are computed as rates, e.g. CPU statistics %USED. A rate is computed based on the refresh interval, the time between successive snapshots. For example, %USED = ( CPU used time at snapshot 2 - CPU used time at snapshot 1 ) / time elapsed between snapshots. The default refresh interval can be changed by the command line option "-d", or the interactive command 's'. The return key can be pressed to force a refresh. In each screen, data is presented at different levels of aggregation. It is possible to drill down to expanded views of this data. Each screen provides different expansion options. It is possible to select all or some fields for which data collection is done. In the case of interactive use of esxtop, the order in which the selected fields are displayed can be selected. For more information, please refer to the official vSphere Monitoring and Performance documentation. You can run esxtop using the ESXi shell. It can only be run by user root. It can be run in three different modes: interactive (default), batch and replay. Worlds in this document refer to ESXi VMkernel schedulable entities, similar to processes in other operating systems. resxtop provides a fine-grain look at resource utilization of a specified server in real time. Resxtop connects to the server using a user name and password. Options --server, --vihost, --portnumber and --username can be used to specify the connection parameters. The resxtop user will be prompted to input the password. If resxtop connects to ESXi directly, --server specifies the ESXi system name. If resxtop connects to a VC server, --server specifies the VC server name and --vihost must be used to specify the ESXi host name registered in the VC server. Resxtop can be run in two different modes in the same way as esxtop: interactive (default) and batch.

Interactive Mode
This is the default mode for esxtop. This mode displays resource usage statistics in the following screens: CPU screen, memory screen, storage adapter screen, storage device screen, storage VM screen, network screen, interrupt screen, power screen, and VSAN screen. A help menu is available for each screen. In this mode there are several command line options available.

  Command-line Options
Prints help for esxtop command-line options. Prints esxtop version number. Tells esxtop to run in secure mode. This disables the interactive command to change delay between screen updates. Show all statistics. This option overides ~/.esxtop60rc configuration file setups and shows all statistics. Load user-defined configuration file. Default configuration file is ~/.esxtop60rc. User can specify the filename to be used as configuration file. User-defined configuration file will be default file name for saving configuration with W command. Specifies the delay between screen updates. Default is 5 seconds. Minimum is 2 seconds. You can change this with the interactive command 's'. If a delay of less than 2 seconds is specified then the delay is set to 2 seconds. Number of iterations. Update the display 'n' number of times and then exit. It is set to 10000 by default.

  Common Statistics Description
Several statistics are displayed on the different screens while esxtop is running in interactive mode. Statistics listed in this section are common across all screens. This first line found at the top of each of the esxtop screens displays the current time, time since last reboot and number of currently running Worlds.
 On this first line CPU load averages are also displayed for the CPU, storage and network screens. The load averages over the past 1, 5 and 15 minutes are displayed. Load averages take into account both running and ready-to-run Worlds. A load average of 1.00 implies that all the physical CPUs are fully accounted for. A load average of 2.00 implies that ESXi may be in need of twice as many physical CPUs as currently available.
 For the memory screen memory overcommit averages are also displayed on the first line. The memory overcommit averages over the past 1, 5 and 15 minutes are displayed. A memory overcommit average of 0.37 implies that the memory is overcommit by 37%. Memory overcommit is the ratio of total requested memory and the "managed memory" minus 1. VMkernel computes the total requested memory as a sum of the following components: (a) VM configured memory (or memory limit setting if set), (b) the user world memory, (c) the reserved overhead memory.

  Common Interactive Commands
Several single-key commands are recognized while esxtop is running in interactive mode. Commands listed in this section are common across all screens. The command to specify the delay between updates is disabled if the s option has been given on the command line. All sorting interactive commands sort in descending order. Displays a help menu for the current screen giving a brief summary of commands, and the status of secure mode. Immediately updates the current screen. Erases and redraws the current screen. Displays a screen to add or remove statistics columns (fields) to or from the current screen. See below for more information. Displays a screen to change order of statistics columns on the current screen. See below for more information. Change the number of statistics rows to show. You will be prompted to enter the number. Any value greater than 0 overrides automatic determination of the number of rows to show, which is based on window size measurement. This number updated on any one of the esxtop screens will be used for all esxtop screens. Change the display length of the name fields, which are "NAME" for the CPU, memory and storage vm screens, "DEVICE" for the sotrage device screen, "DNAME" for the network screen, "DEVICES" for the interrupt screen. You will be prompted to enter the number. Any value greater than 0 overrides the default number of characters of the name field to show. A value equals to 0 means the default value applies. Change the delay between updates. You will be prompted to enter the delay time, in seconds, between updates. Fractional values are recognized down to microseconds. The default value is 5 seconds. The minimum value is 2 seconds. This command is not available in secure mode. Write current setup to ~/.esxtop60rc, or to user-defined configuration file specified by -c option. This is the recommended way to write an esxtop configuration file. See below for more information. Kill a world. (This command is only available in esxtop, not in resxtop.) Quit. Switch to CPU resource utilization screen. Switch to memory resource utilization screen. Switch to storage (disk) adapter resource utilization screen. Switch to storage (disk) device resource utilization screen. Switch to storage (disk) VM resource utilization screen. Switch to network resource utilization screen. Switch to interrupt utilization screen. Switch to power utilization screen. Switch to VSAN utilization screen.

  The Statistics Columns (fields) and Order Screens
After pressing f, F, o or O you will be shown a screen specifying the field order on the top line and short descriptions of the field contents. The field order string uses the following syntax: if the letter in the field string corresponding to a field is upper case, the field will be displayed. This is also indicated by an asterisk in front of the field description. The order of the fields corresponds to the order of the letters in the string. From the field select screen you can toggle the display of a field by pressing the corresponding letter. From the order screen you may move a field to the left by pressing the corresponding upper case letter and to the right by pressing the lower case one.

  Configuration File
esxtop reads its default configuration from ~/.esxtop60rc or from specified file name with -c option. This configuration file contains ten lines. The first nine lines contain lower and upper case letters to specify which fields in what order are to be displayed on the CPU, memory, storage (disk) adapter, storage (disk) device, storage (disk) VM, network, interrupt, power and VSAN screens. The letters correspond to the letters in the Fields or Order screens for the respective esxtop screens. Editing this file by hand is not recommended. Select fields and order in a running esxtop process and save this file using the W interactive command. The last line contains information on the other options. Most important, if you have saved a configuration in secure mode, you will not get an insecure esxtop without removing the lower 's' from this last line of your ~/.esxtop60rc. A digit specifies the delay time between updates. As in interactive mode, a lower 'c', 'm', 'd', 'u', 'v', 'n', 'i', 'p' and 'x' determines the screen esxtop starts on.

  CPU Screen
This screen displays server-wide and per Resource Pool/World CPU utilization statistics. By default, World CPU utilization statistics are aggregated into Resource Pools of Worlds. For Worlds belonging to a Virtual Machine, these Resource Pools correspond to running Virtual Machines. All other Worlds are logically aggregated into Resource Pools to save vertical window space.

Statistics Descriptions
This line displays the percentage CPU usage per PCPU, and the percentage CPU usage averaged over all PCPUs. A PCPU refers to a physical hardware execution context -- a physical CPU core if hyper-threading is unavailable or disabled, or a logical CPU (aka LCPU or SMT thread) if hyper-threading is enabled. PCPU USED(%) is the percentage of PCPU nominal frequency that was used since the last screen update. PCPU USED(%) is the total sum of %USED for Worlds that ran on this PCPU. Note that if PCPU is running at frequency that is higher than its nominal (rated) frequency, then PCPU USED(%) can be greater than 100%. Also, note that if a PCPU and its partner are busy when hyper-threading is enabled, then each PCPU is accounted for half of the CPU usage. This line displays the percentage CPU utilization per PCPU, and the percentage CPU utilization averaged over all PCPUs. A PCPU refers to a physical hardware execution context -- a physical CPU core if hyper-threading is unavailable or disabled, or a logical CPU (aka LCPU or SMT thread) if hyper-threading is enabled. UTIL(%) represents the raw PCPU utilization, that is the percentage of real time that PCPU was not idle. USED(%) and UTIL(%) may be different due to power management technologies or hyper-threading. This line displays the percentage utilization of each core and the average over all cores. It is displayed only when hyper-threading is used. In interactive mode, the percentage utilization of a core is aligned with the percentage utilization of its first logical CPU (aka LCPU or SMT thread). A core is utilized, if either or both of its logical CPUs are utilized. The percentage utilization of a core is not the sum of the percentage utilization of both logical CPUs on this core. This line displays the percentages of total CPU time as reported by the ESXi Service Console. us is for percentage user time, sy is for percentage system time, id is for percentage idle time and wa is for percentage wait time. cs/sec is for the context switches per second recorded by the ESXi Service Console. The Resource Pool ID of the running World's Resource Pool or World ID of running World. The Resource Pool ID of running World's Resource Pool. The Leader World ID of running World's Resource Pool. The name of running World's Resource Pool or name of running World. The number of worlds in running World's Resource Pool. If a Resource Pool is expanded using the interactive command e (see interactive commands below) then NWLD for all the resulting Worlds belonging to the Resource Pool will be 1 (some Resource Pool like the console Resource Pool have only 1 member). This is a set of CPU statistics made up of the following percentages. Note that for a World, the percentages are a percentage of one physical CPU core. The percentage of physical CPU core cycles used by the Resource Pool/World. %USED may depend on the frequency with which CPU core is running. When running with lower CPU core frequency, %USED can be smaller than %RUN. On CPUs which support turbo mode, CPU frequency can also be higher than nominal (rated) frequency, and in that case %USED can be larger than %RUN. The percentage of time spent in the ESXi VMkernel on behalf of the Resource Pool/World to process interrupts and to perform other system activities. This time is part of used time used to calculate %USED above. The Percentage of system time that was spent on behalf of some other Resource Pool/World while Resource Pool/World was scheduled. This time is not included in the %SYS for Resource Pool/World. For example, if Resource Pool A was currently scheduled and a network packet for Resource Pool B was being processed by the ESXi VMkernel, then the time spent doing so appears as %OVRLP for Resource Pool A and %SYS for Resource Pool B. Percentage of total time scheduled. This time does not account for hyper-threading and system time. Hence, on a hyper-threading enabled server, the %RUN can be twice as large as %USED. The total percentage of time the Resource Pool/World spent in wait state. I.e., the world is waiting for some VMkernel resource. Note that this percentage includes the percentage of time the Resource Pool/World was idle. The total percentage of time the Resource Pool/World spent in blocked state waiting for events. The percentage of time the Resource Pool/World spent in ready, co-deschedule state. This statistic is for internal to VMware use only. The percentage of time the Resource Pool/World was idle. Subtracting this percentage from %WAIT above gives you the percentage of time the Resource Pool/World was waiting on some event. The percentage of time the Resource Pool/World was ready to run. Percentage of time the ESXi VMkernel deliberately did not run the Resource Pool/World because that would violate the Resource Pool/World's limit setting. Since the Resource Pool/World is ready to run when it is prevented from running in this way, the %MLMTD (max limited) time is included in %RDY time. Percentage of time the Resource Pool/World was waiting for the ESXi VMkernel swapping memory. The %SWPWT (swap wait) time is included in the %WAIT time. This is a set of CPU statistics made up of the following, per second, event rates. These statistics are for internal to VMware use only. This is a set of CPU statistics made up of the following CPU allocation configuration parameters. Resource Pool/World attribute reservation. Resource Pool/World attribute limit. Resource pool attribute shares. This is a set of CPU statistics made up of the following CPU configuration parameters and statistics. The percentage of time the Resource Pool/World was ready to run but was not scheduled to run because of cpu resource contention. The percentage of time the Resource Pool/World was ready to run but was not scheduled to run because of memory resource contention. The CPU demand in percentage. It represents the average active CPU load in the past 1 minute. The Effective Min in MHz for the Resource Pool/World. The amount of CPU resources guaranteed to the world if all the worlds on the system start contending for CPU resources. ESXi VMkernel dynamically calculates the EMIN value for all worlds based on the resource settings (Reservations, Limits and Shares) of all the resource pools and VMs on a system. This statistic is for internal to VMware use only. The timer rate the World is currently requesting for. Bit mask showing the current scheduling affinity for the World. The physical or logical processor the World was found to be running on. current hyper-threading configuration. World is currently quaratined or not. 'N' implies no and 'Y' implies yes. The current CPU power consumption in Watts accounted for a Resource Pool.

Interactive Commands
Expand/Rollup Resource Pool/Worlds CPU statistics. Allows viewing CPU resource utilization statistics broken down by individual Worlds belonging to a Resource Pool. Sort Resource Pools/Worlds by Resource Pool %USED column. This is the default sort order. Sort Resource Pools/Worlds by Resource Pool %RDY column. Sort Resource Pools/Worlds by GID column. Display VM instances only.

  Memory Screen
This screen displays server-wide and per Resource Pool memory utilization statistics. As on the CPU screen, Resource Pools correspond to running Virtual Machines or Worlds that consume memory. Note the distinction between machine memory and physical memory in this section. Physical memory is the virtual hardware physical memory presented to the VM. Machine memory is actual physical RAM in the ESXi host.

Statistics Descriptions
This line displays the machine memory statistics for the server. All numbers are in megabytes. total is for the total amount of machine memory in the server, cos is for the amount of machine memory allocated to the ESXi Service Console, vmk is for the amount of machine memory being used by the ESXi VMkernel, other is for the amount of machine memory being used by everything other than the ESXi Service Console and ESXi VMkernel and free is for the amount of machine memory that is free. This line displays the machine memory statistics for the ESXi Server VMkernel. All numbers are in megabytes. managed is for the total amount of machine memory managed by the ESXi Server VMkernel, minfree is for the minimum amount of machine memory that the ESXi VMkernel would like to keep free, rsvd is for the amount of machine memory that is currently reserved, ursvd is for the amount of machine memory that is currently unreserved and state is for the memory state as reported by the ESXi VMkernel. Possible values are high, soft, hard and low. Here, high implies that the machine memory is not under any pressure and low implies that the machine memory is under pressure. While the host's memory state is not used to determine whether memory should be reclaimed from VMs (that decision is made at the resource pool level), it can affect what mechanisms are used to reclaim memory if necessary. In the high and soft states, ballooning is favored over swapping. In the hard and low states, swapping is favored over ballooning. This line displays the memory statistics as reported by the ESXi Service Console. All numbers are in megabytes. free is for the amount of idle machine memory, swap_t is for the total swap configured, swap_f is for the amount of swap free, r/s is for the rate at which memory is swapped in from disk and w/s is for the rate at which memory is swapped to disk. This line displays the ESXi NUMA statistics. This line is only displayed if ESXi is running on a NUMA server. All numbers are in megabytes. For each NUMA node in the server there are two statistics that are displayed. The first statistic is the total amount of machine memory in the NUMA node that is managed by ESXi. The second statistic, that is displayed within round brackets, is the amount of machine memory in the node that is currently free. This line displays the ESXi page-sharing statistics. All numbers are in megabytes. shared is for the amount of physical memory that is being shared, common is for the amount of machine memory that is common across World(s) and saving is for the amount of machine memory that is saved due to page-sharing. The monitor maps guest physical memory to machine memory. VMkernel selects to map guest physical pages with the same content to the same machine page. In other words, those guest physical pages are sharing the same machine page. This kind of sharing can happen within the same VM or among the VMs. Since each VM's "shared" memory measures guest physical memory, the host's "shared" memory may be larger than the total amount of machine memory if memory is overcommitted. "saving" illustrates the effectiveness of page sharing for saving machine memory. This line displays the ESXi swap usage statistics. All numbers are in megabytes. curr is for the current swap usage, rclmtgt is for where ESXi expects the relaimed memory using swapping and compression to be, r/s is for the rate at which memory is swapped in by ESXi from disk and w/s is for the rate at which memory is swapped to disk by the ESXi Server. Note that rclmtgt is the total target for both swapping and compression. We should compare it against the sum of the curr swapped memory and the saved memory due to memory compression shown in the ZIP line. This line displays the ESXi memory compression statistics. All numbers are in megabytes. zipped is for the total compressed physical memory, saved is for the saved memory by compression. This line displays the memory balloon statistics. All numbers are in megabytes. curr is for the total amount of physical memory reclaimed using the vmmemctl modules, target is for the total amount of physical memory ESXi would like to reclaim using the vmmemctl modules and max is for the maximum amount of physical memory ESXi can reclaim using the vmmemctl modules. Resource Pool ID The Leader World ID of Resource Pool. Resource Pool name Resource Pool attribute reservation. Resource Pool attribute limit. Resource Pool attribute shares. Current Home Node for Resource Pool. This statistic is only applicable on NUMA systems. Number of NUMA migrations between two snapshots. It includes balance migration, inter-node VM swaps performed for locality balancing and load balancing. This statistic is only applicable on NUMA systems. Current amount of remote memory being accessed by Resource Pool. This statistic is only applicable on NUMA systems. Current amount of local memory being accessed by Resource Pool. This statistic is only applicable on NUMA systems. Current percentage memory being accessed by Resource Pool that is local. This statistic is only applicable on NUMA systems. The guest memory being allocated for Resource Pool on NUMA node x. "x" is the node number. This statistic is only applicable on NUMA systems. The VMM overhead memory being allocated for Resource Pool on NUMA node x. "x" is the node number. This statistic is only applicable on NUMA systems. The amount of physical memory allocated to a Resource Pool. The amount of physical memory granted to a Resource Pool. It is the mapped guest physical memory. GRANT - SHRDSVD is the consumed host machine memory. For a user world, it is the amount of virtual memory that is backed by machine memory. The amount of machine memory the ESXi VMkernel wants to allocate to Resource Pool. It includes the overhead memory for a VM. This is an internal counter, which is computed by ESXi memory scheduler. Usually, there is no need to worry about this. Roughly speaking, SZTGT of all the VMs is computed based on the resource usage, available memory, and the "limit/reservation/shares" settings. This computed SZTGT is compared against the current memory consumption plus overhead memory for a VM to determine the swap and balloon target, so that VMkernel may balloon or swap appropriate amount of memory to meet its memory demand. The working set estimate for Resource Pool. VMkernel estimates active memory usage for a VM by sampling a random subset of the VM's memory resident in machine memory to detect the number of memory reads and writes. VMkernel then scales this number by the size of VM's configured memory and averages it with previous samples. Over time, this average will approximate the amount of active memory for the VM. Note that ballooned memory is considered inactive, so, it is excluded from TCHD. The write working set estimate for Resource Pool. Percentage of guest physical memory that is being referenced by the guest. This is an instantaneous value. Percentage of guest physical memory that is being referenced by the guest. This is a slow moving average. Percentage of guest physical memory that is being referenced by the guest. This is a fast moving average. Percentage of guest physical memory that will be referenced by the guest. This is an estimation. This statistic is for internal to VMware use only. memory balloon driver is installed or not. 'N' implies no and 'Y' imples yes. The amount of physical memory reclaimed from Resource Pool by way of ballooning. This can be called "balloon size". A large MCTLSZ
 means lots of this VM's guest physical memory is "stolen" to decrease host memory pressure. This usually is not a problem, because balloon driver tends to smartly steal guest physical memory that cause little performance problems. The amount of physical memory ESXi would like to reclaim from Resource Pool by way of ballooning. This is an internal counter, which is computed by ESXi memory scheduler. Usually, there is no need to worry about this. The maximum amount of physical memory ESXi can reclaim from Resource Pool by way of ballooning. This maximum is dependent on guest operating system type. Current swap usage by Resource Pool. Where ESXi expectd the Swap usage by the Resource Pool to be. Rate at which memory is being swapped in by ESXi from disk for the Resource Pool. When a VM is requesting machine memory to back its guest physical memory that was swapped out to disk, VMkernel reads in the page. The swap-in operation is synchronous. Rate at which Resource Pool memory is being swapped to disk by ESXi. This stats refers to the VMkernel swapping, not the guest swapping. Rate at which memory is faulted from host cache. Rate at which memory is written to the host cache from various sources. Amount of data read from checkpoint file. Size of checkpoint file. Resource Pool physical pages that are zeroed. Resource Pool physical pages that are shared. VMkernel page sharing module scans and finds guest physical pages with the same content and backs them with the same machine page. It accounts the total guest physical pages that are shared by the page sharing module. Machine pages that are saved due to Resource Pool shared pages. Because a machine page is shared by multiple guest physical pages, we only charge "1/ref" page as the consumed machine memory for each of the guest physical pages, where "ref" is the number of references. So, the saved machine memory will be "1 - 1/ref" page. This statistic is for internal to VMware use only. Current space overhead for the user world. This statistic is for internal to VMware use only. Current space overhead for Resource Pool. It includes the overhead memory consumed by the monitor, the VMkernel and the vmx user world. Maximum space overhead that may be incurred by Resource Pool. It is the overhead memory a VM wants to consume in the future. This amount of reserved overhead memory includes the overhead memory reserved by the monitor, the VMkernel, and the vmx user world. This is a set of memory statistics made up of minimum memory commitment target, current memory commitment target, charged memory commitment, and pages per share. These statistics are for internal to VMware use only. Compression memory cache size. Used compression memory cache. Compressed memory per second. Decompressed memory per second.

Interactive Commands
Display VM instances only. Sort Resource Pools by Resource Pool MEMSZ column. This is the default sort order. Sort Resource Pools by Resource Pool MCTLSZ column. Sort Resource Pools by GID column.

  Storage Adapter Screen
This screen displays server-wide storage utilization statistics. On this screen statistics are aggregated per storage adapter by default. Statistics can also be viewed per storage path. See below for more information.

Statistics Descriptions
The name of the storage adapter. The storage path name. This name is only visible if the corresponding adapter is expanded. See interactive command 'e' below. When the statistics are expanded to path level, esxtop shows the path statistics. The number of paths. The storage adapter queue depth. This is the maximum number of ESXi VMkernel active commands that the adapter driver is configured to support. The number of commands completed per second. The number of read commands completed per second. The number of write commands completed per second. The megabytes read per second. The megabytes written per second. The number of SCSI reservations per second. The number of SCSI reservation conflicts per second. The average device latency (millisecs) per command. It is a good indicator of performance of the backend storage. If IO latencies are suspected to be causing performance problems, DAVG should be examined. Compare IO latencies with corresponding data from the storage array. If they are close, check the array for misconfiguration or faults. If not, compare DAVG with corresponding data from points in between the array and the ESXi Server, e.g., FC switches. If this intermediate data also matches DAVG values, it is likely that the storage is under-configured for the application. Adding disk spindles or changing the RAID level may help in such cases. The average ESXi VMkernel latency (millisecs) per command. It's value should be very small in comparison to the DAVG value and should be close to zero. When there is a lot of queuing in ESXi, KAVG can be as high, or even higher than DAVG. The average Guest OS latency (millisecs) per command. The average queue latency (millisecs) per command. Response time is the sum of the time spent in queues in the storage stack and the service time spent by each resource in servicing the request. The largest component of the service time is the time spent in retrieving data from physical storage. If QAVG is high, another line of investigation is to examine the queue depths at each level in the storage stack. The average device read latency (millisecs) per read. The average ESXi VMkernel read latency (millisecs) per read. The average Guest OS read latency (millisecs) per read. The average queue read latency (millisecs) per read. The average device write latency (millisecs) per write. The average ESXi VMkernel write latency (millisecs) per write. The average Guest OS write latency (millisecs) per write. The average queue write latency (millisecs) per write. Failed commands per second. Failed read commands per second. Failed write commands per second. The megabytes read per second by failed commands. The megabytes written per second by failed commands. The number of reserve commands failed per second. The number of commands aborted per second. It can indicate that the storage system is unable to meet the demands of the guest operating system. Abort commands are issued by the guest when the storage system has not responded within an acceptable amount of time, e.g. 60 seconds on some windows OS's. Also, resets issued by a guest OS on its virtual SCSI adapter will be translated to aborts of all the commands outstanding on that virtual SCSI adapter. If multiple paths are connected to the same lun, this statistic is per path when it is expanded to luns. The number of commands reset per second. The number of PAE commands per second. It may point to hardware misconfiguration. When the guest allocates a buffer, the vmkernel assigns some machine memory, which might come from a .highmem. region. If you have a driver that is not PAE-aware, then this counter is updated if accesses to this memory region result in copies by the vmkernel into a lower memory location before issuing the request to the adapter. This might happen if you do not populate the DIMMs with low memory first, then you may artificially cause .highmem. memory accesses. The number of PAE copies per second. The number of split commands per second. Commands can be split when they reach the vmkernel. This might impact perceived latency to the guest. The guest may be issuing commands of large block sizes which have to be broken down by the vmkernel. Splitting can also occur when IOs fall across partition boundaries but these are easily differentiated from the splitting as a result of the IO size. The number of split copies per second.

Interactive Commands
Expand/Rollup storage adapter statistics. Allows viewing storage resource utilization statistics broken down by individual paths belonging to an expanded storage adapter. You will be prompted to enter the adapter name. Sort by READS/s column. Sort by WRITES/s column. Sort by MBREAD/s column. Sort by MBWRTN/s column. Sort first by ADAPTER column, then by PATH column. This is the default sort order.

  Storage Device Screen
This screen displays server-wide storage utilization statistics. On this screen statistics are aggregated per storage device by default. Statistics can also be viewed per path, world, or partition. See below for more information.

Statistics Descriptions
The name of the storage device. The path name. This name is only visible if the corresponding device is expanded to paths. See interactive command 'p' below. The world id. This id is only visible if the corresponding device is expanded to worlds. See interactive command 'e' below. The world statistics are per world per device. The partition id. This id is only visible if the corresponding device is expanded to partitions. See interactive command 't' below. The number of paths. The number of worlds. The number of partitions. The number of shares. This statistic is only applicable to worlds. The block size in bytes. The number of blocks of the device. The storage device queue depth. This is the maximum number of ESXi VMkernel active commands that the device is configured to support. The World queue depth. This is the maximum number of ESXi VMkernel active commands that the World is allowed to have. Note that this is a per device maximum for the World. It is valid only if the corresponding device is expanded to worlds. The number of commands in the ESXi VMkernel that are currently active. This statistic is only applicable to worlds and devices. The number of commands in the ESXi VMkernel that are currently queued. This statistic is only applicable to worlds and devices. Queued commands are commands waiting for an open slot in the queue. A large number of queued commands may be an indication that the storage system is overloaded. A sustained high value for the QUED counter signals a storage bottleneck which may be alleviated by increasing the queue depth. Check that LOAD < 1 after increasing the queue depth. This should also be accompanied by improved performance in terms of increased cmd/s. The percentage of queue depth used by ESXi VMkernel active commands. This statistic is only applicable to worlds and devices. %USD is a measure of how many of the available command queue "slots" are in use. Sustained high values indicate the potential for queueing; you may need to adjust the queue depths for system's HBAs if QUED is also found to be consistently > 1 at the same time. Queue sizes can be adjusted in a few places in the IO path and can be used to alleviate performance problems related to latency. The ratio of ESXi VMkernel active commands plus ESXi VMkernel queued commands to queue depth. This statistic is only applicable to worlds and devices. The number of commands issued per second. The number of read commands issued per second. The number of write commands issued per second. The megabytes read per second. The megabytes written per second. The number of SCSI reservations per second. The number of SCSI reservation conflicts per second. The average device latency (millisecs) per command. The average ESXi VMkernel latency (millisecs) per command. The average Guest OS latency (millisecs) per command. The average queue latency (millisecs) per command. The average device read latency (millisecs) per read. The average ESXi VMkernel read latency (millisecs) per read. The average Guest OS read latency (millisecs) per read. The average queue read latency (millisecs) per read. The average device write latency (millisecs) per write. The average ESXi VMkernel write latency (millisecs) per write. The average Guest OS write latency (millisecs) per write. The average queue write latency (millisecs) per write. The number of commands aborted per second. The number of commands reset per second. The number of PAE commands per second. This statistic is only applicable to paths. The number of PAE copies per second. This statistic is only applicable to paths. The number of split commands per second. This statistic is only applicable to paths. The number of split copies per second. This statistic is only applicable to paths. The number of clone read commands. The number of clone write commands. The number of failed clone commands. The megabytes clone read per second. The megabytes clone written per second. The number of ats commands. The number of failed ats commands. The number of zero commands. The number of failed zero commands. The megabytes zeroed per second. The average clone latency (millisecs) per successful command. The average clone latency (millisecs) per failed command. The average ats latency (millisecs) per successful command. The average ats latency (millisecs) per failed command. The average zero latency (millisecs) per successful command. The average zero latency (millisecs) per failed command.

Interactive Commands
Expand/Rollup storage world statistics. Allows viewing storage resource utilization statistics broken down by individual worlds belonging to an expanded storage device. You will be prompted to enter the device name. The statistics are per world per device. Expand/Rollup storage path statistics. Allows viewing storage resource utilization statistics broken down by individual paths belonging to an expanded storage device. You will be prompted to enter the device name. Expand/Rollup storage partition statistics. Allows viewing storage resource utilization statistics broken down by individual partitions belonging to an expanded storage device. You will be prompted to enter the device name. Sort by READS/s column. Sort by WRITES/s column. Sort by MBREAD/s column. Sort by MBWRTN/s column. Sort first by DEVICE column, then by PATH/WORLD/PARTITION column. This is the default sort order.

  Storage VM Screen
This screen displays VM-centric storage statistics. On this screen, statistics are aggregated on a per-resource-pool basis by default. One VM has one corresponding resource pool, so, they are equivalent to per-VM statistics. Statistics can also be viewed on a per-vscsi-device basis.

Statistics Descriptions
The Resource Pool ID of the Resource Pool or Vscsi ID of Vscsi Device. The Resource Pool ID of Resource Pool. The name of Resource Pool. The name of the Virtual Device. The number of vscsi devices. The number of commands issued per second. The number of read commands issued per second. The number of write commands issued per second. The megabytes read per second. The megabytes written per second. The average latency (millisecs) per read. The average latency (millisecs) per write.

Interactive Commands
Expand/Rollup storage vscsi device statistics. Allows viewing storage resource utilization statistics broken down by individual vscsi devices belonging to a group. You will be prompted to enter the group id. The statistics are per vscsi device. Sort by READS/s column. Sort by WRITES/s column. Sort by MBREAD/s column. Sort by MBWRTN/s column. Sort first by VM column, then by VSCSI column. This is the default sort order.

  Network Screen
This screen displays server-wide network utilization statistics. On this screen statistics are arranged per port per configured virtual network device. For physical NIC statistics, see the row corresponding to the port that the physical NIC is connected to. For virtual NIC configured in a particular Virtual Machine statistics, see the row corresponding to the port that the virtual NIC is connected to.

Statistics Descriptions
The virtual network device port id. Y implies the corresponding port is an uplink. N implies it is not. Y implies the corresponding link is up. N implies it is not. The link speed in MegaBits per second. Y implies the corresponding link is operating at full duplex. N implies it is not. The virtual network device port user. The physical NIC name for the team uplink. The virtual network device name. The number of packets transmitted per second. The number of packets received per second. The MegaBits transmitted per second. The MegaBits received per second. The average packet size transmitted in bytes. The average packet size received in bytes. The percentage of transmit packets dropped. The percentage of receive packets dropped. Number of actions per second. This statistic is for internal to VMware use only. The number of multicast packets transmitted per second. The number of multicast packets received per second. The number of broadcast packets transmitted per second. The number of broadcast packets received per second.

Interactive Commands
Sort by MbTX/s column. Sort by MbRX/s column. Sort by PKTTX/s column. Sort by PKTRX/s column. Sort by PORT ID column. This is the default sort order.

  Interrupt Screen
This screen displays interrupt utilization statistics. On this screen statistics are arranged per interrupt cookie.

Statistics Descriptions
The interrupt cookie. The total number of interrupts per second across all the CPUs. It measures how often an interrupt is raised on the device. E.g., If you have 2 CPUs, COUNT/s = COUNT_0 + COUNT_1. COUNT_0 and COUNT_1 are described below. The number of interrupts per second on CPU 'x'. This is a per CPU counter. Comparing "COUNT_x" for the same interrupt cookie on different CPUs can tell us how balanced the interrupts are scheduled across all the CPUs. The average processing time in microseconds per interrupt. It is the average for all the interrupts of the same cookie. The average processing time in microseconds per interrupt on CPU 'x'. It averages only the interrupts raised on CPU 'x'. The devices that use the interrupt cookie. If the interrupt cookie is not enabled for the device, its name is enclosed in "<>", e.g. "<VMK device>".

  Power Screen
This screen displays CPU Power utilization statistics. On this screen statistics are arranged per PCPU. A PCPU refers to a physical hardware execution context -- a physical CPU core if hyper-threading is unavailable or disabled, or a logical CPU (aka LCPU or SMT thread) if hyper-threading is enabled.

Statistics Descriptions
The current total power usage in Watts. The total power cap in Watts. The percentage of PCPU nominal frequency that was used since the last screen update. It is the same as PCPU USED(%) shown in the CPU Screen. The raw PCPU utilization, that is the percentage of real time that PCPU was not idle. It is the same as PCPU UTIL(%) shown in the CPU Screen. The percentage of time the PCPU spent in C-State 'x'. The percentage of time the PCPU spent in P-State 'x'. The percentage of time the PCPU spent in T-State 'x'.

  VSAN Screen
This screen displays VSAN statistics.

Statistics Descriptions
Whether VSAN is currently enabled on the host. The name of VSAN DOM Role. Number of read operations completed per second. Megabytes read per second. Number of write operations completed per second. Megabytes written per second. Number of recovery write operations completed per second. Megabytes written for recovery per second. Number of unmap operations completed per second. Gigabytes unmapped per second. Number of recovery unmap operations completed per second. Gigabytes unmapped for recovery per second. Standard deviation of latency in millisecs for read, write, recovery write, unmap and recovery unmap. Average latency in millisecs for read, write, recovery write, unmap and recovery unmap.

Batch Mode
Allows collecting and saving resource utilization statistics in a file. Running in this mode is a two step process.
 Step one requires running esxtop in interactive mode, switching to each of the nine available screens, selecting the columns on each screen you are interested in and saving this configuration in the ~/.esxtop60rc file using the W interactive command. See Interactive Mode above. Note that the global stats are always available in batch mode.
 Step two requires running esxtop in batch mode and redirecting the output to a file. This may be done as follows: esxtop -b > foo.csv. Note that the file name needs to have a .csv extension. Esxtop does not enforce this, but post-processing tools mentioned next require this.
 Statistics collected in batch mode may be post-processed using such MS Windows applications like Excel and Perfmon.
 In this mode, esxtop will not accept interactive commands. Esxtop in batch mode runs until it produces the number of iterations requested (see command-line option n below for more details) or until killed using CTRL c.
 In this mode there are several command line options available.

  Command-line Options
Run esxtop in Batch mode. Specifies the delay between statistics snapshots. Default is 5 seconds. Minimum is 2 seconds. If a delay of less than 2 seconds is specified then the delay is set to 2 seconds. Number of iterations. Collect and save statistics this number of times and then exit.

Replay Mode
Replays resource utilization statistics collected using vm-support (see vm-support man page for more information), resxtop does not support replay mode. Running in this mode is a two step process.
 Step one requires running vm-support in snapshot mode on the ESXi Service Console. This may be done as follows: vm-support -S -d duration -i interval. The resulting zipped tar file needs to be un-zipped and un-tar'ed before use by esxtop replay mode.
 Step two requires running esxtop in replay mode. This may be done as follows: esxtop -R vm-support_dir_path. Note that it is not required to run esxtop replay mode on the ESXi Service Console.
 In replay mode, esxtop needs to be of the same verison as that of ESXi to avoid compatibility problem.
 Replay mode can be run to produce Batch Mode style output (see command-line option b below for more information).
 In replay mode, esxtop accepts the same set of interactive commands as in Interactive Mode. Esxtop in replay mode runs until there are no more vm-support collected snapshots to be read or esxtop produces the number of iterations requested (see command-line option n below for more details).
 In this mode there are several command line options available.

  Command-line Options
Specifies the path to the vm-support collected snapshot's directory. Run esxtop in Batch mode. Specifies the delay between screen updates. Default is 5 seconds. Minimum is 2 seconds. If a delay of less than 2 seconds is specified then the delay is set to 2 seconds. Number of iterations. Update the display this number of times and then exit.

EXPERIMENTAL FEATURES

  Export and Import Entities
This feature allows users show only the entities that they are interested in, such as groups, devices, adapters, ports, interrupt cookies, etc. The ability of selecting interesting entities can also reduce the CPU load of esxtop itself. Two command line options are introduced for this purpose: [export-entity entity-file] and [import-entity entity-file]. Users can do three steps to select entities which they want to display. (1) Users first export the list of current entities using the export-entity option, by running esxtop -export-entity entity-file . The entity-file is generated, including the ids of all the existing entities at that time. (2) Then, they can edit the generated entity-file to delete the entities by inserting # before their names. (3) At last step, users can import entity-file using the import-entity option, by running esxtop -import-entity entity-file . Therefore, esxtop will only show the stats for the entities selected in the entity-file. Note that these two options are only available in esxtop, not in resxtop.

  Entity Highlight/Expand/Deletion Interactive Operations
Under interactive mode, users can use keys in numpad to highlight, expand and delete entities. User can press '8' to move up the highlight cursor, and '2' to move down the highlight cursor. Users can press '6' to can expand/unroll the selected entity when appropriate. In CPU Screen, it expands/rollups Resource Pool/Worlds CPU statistics. In Storage Device Screen, pressing '6' can rotate the three different expanding modes and rollup modes. Pressing '6' the first time expands storage world statistics; pressing '6' the second time expands storage path statistics; pressing '6' the third time expands storage partition statistics; pressing '6' the fourth time rollups to the default view. In Disk VM Screen, pressing '6' can expand/rollup Resource Pool/Worlds Disk statistics. Users can press '4' to remove the highlighted entity from display, which will decrease the amount of stats we try to collect. (Note that this delete operation is only available in esxtop, not in resxtop.) Users can press '.' to restore all the entities and we will collect data for all of them again.

  NOTES:
Note that the disk VM screen shows the storage stats for a VM. The selection of group entities and disk device entities have effect to this screen. If the group is not chosen, it will not show up. If the device is not chosen, all operations on this device will be counted as zeros.

FILES
~/.esxtop60rc The personal configuration file.

SEE ALSO
vm-support(1), cpu(8), mem(8), diskbw(8), numa(8)

COPYRIGHT
VMware ESXi is Copyright 1998-2014 VMware, Inc. All rights reserved.

DISCLAIMER
Based on top source code from the OpenBSD distribution.
   Copyright (c) 1997, Jason Downs. All rights reserved.
   Copyright (c) 1984, 1989, William LeFebvre, Rice University
   Copyright (c) 1989, 1990, 1992, William LeFebvre,
                        Northwestern University Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.