Objective 6.1 of VCAP6-Deploy exam covers following topics:

• Configure esxtop / resxtop custom profiles
• Evaluate use cases for and apply esxtop / resxtop Interactive, Batch and Replay modes
• Use esxtop / resxtop to collect performance data
• Given esxtop / resxtop output, identify relative performance data for capacity planning purposes

Before starting discussing on these topics, I want to cover a few basics of vSphere Management Assistant (vMA) as we will be using it for performing few tasks listed in this objective.

What is vSphere Management Assistant (vMA)?

The vSphere Management Assistant (vMA) is a virtual machine that includes prepackaged software such as a Linux distribution, the vSphere command‐line interface, and the vSphere SDK for Perl. Basically it is the missing service console for ESXi. But it’s more than that too.

This allows administrators to run scripts or agents that interact with ESX/ESXi and vCenter Server systems without having to explicitly authenticate each time. vMA can also collect ESX/ESXi and vCenter Server logs and store the information for analysis.

vMA 6.0 can be downloaded from here. It is available in the form of .ova file and installation is pretty much straight forward like any other VMware appliance and I am not covering installation steps. Basic installation/configuration steps can be read from this article. 

Getting started with vMA

SSH to vMA appliance and login with user ‘vi-admin’ and password set during deployment. The first task is to add remote servers to vMA so that we can execute commands on those servers without logging onto them.

1: Adding a remote server

2: List Servers : Once you have added all your remote servers, you can list them by executing below command

3: Set Remote Server : To execute command on a remote server which is added to vMA previously, we have to first set it as target server 

4: Add Esxi SSH Thumbprint: When an Esxi host is set as target server, and you try to execute a command against that server, you will see a message about connection failed as the server thumbprint is not trusted. We have to add  SSH thumbprint of target server on using credstore_admin.pl utility

5: Execute command : Once a thumbprint is added, you can execute command on the added target server

Now since basics of vMA has been covered, lets move on to main topics of this objective.

                                          Configure esxtop / resxtop custom profiles

To gather performance data and monitor an ESXi host, following 2 tools can be used:

• esxtop. This command runs locally on an Esxi host and  provides real time CPU, memory, disk and network data for hosts and virtual machines.
• resxtop. This is a remote version of esxtop. It is included as part of vCLI and is present on the vMA.

By default when you run esxtop command, you are presented with cpu utilization related stats of an Esxi host, but esxtop has many different displays and options around which you can play. 

When running esxtop simply select ‘h’ to see the options available. Typically you will see the following options on your screen

Now suppose you want to see network related statistics of Esxi host, just press ‘n’ and the view will be changed

If you wish to exclude some fields which you do not want to monitor, then press ‘f’ button and press the letter to toggle the field. 

For example if I do not want to Include field MegaBits Rx/s, then I will press ‘K’ and if I want to include Multicast packets, I will press ‘P’ and hit enter. You will see 2 new fields added in the network statistics.

There are lot and lot of options that can be used with esxtop. This article covers a full list of available options.

Running resxtop

As we discussed earlier, resxtop is available with vMA. To run resxtop against an Esxi host, fire below command

and it will show you cpu statistics of the host by default. You can toggle the fields which you want to see and can create custom profiles and save it so that next time when you run that profile, you will see only that data which you need.   

To save the current metrics into a profile we can save by using a W, and then naming the profile. For example I am saving esxi01 cpu statistics in a file named esxi01-cpu. 

Now when you want to run resxtop against this host and want to see the custom fields, you don’t have to toggle the options. Just run the profile which you saved earlier 

Evaluate use cases for and apply esxtop / resxtop Interactive, Batch and Replay modes

There are 3 modes in which esxtop/resxtop can be used. These modes are: 

• Interactive: This is default mode in which statistics are displayed in real time in different panels.
• Batch: In batch mode, resxtop (or esxtop) does not accept interactive commands. This mode is used for collecting statistics over a set period of time. Further analyisis is performed retrospectively using several different tools including Microsoft Excel, ESXlot and Perfmon.
• Replay: In replay mode, esxtop replays resource utilization statistics collected using vm-support. We can view the captured performance information for a particular duration or time period as like real time to view what was happened during that time. 

Running esxtop/resxtop in Batch mode

To collect esxtop/resxtop data in batch mode, fire a command similar to: 

where:

• b = batch mode
• a = all of the statistics 
• d = delay in seconds between statistics snapshots
• n = number statistic snapshots saved before batch is stopped.

Running esxtop in Replay mode

To run esxtop in replay mode, login to Esxi host via root user over SSH session and use a command similar to as shown below, to capture performance statistics:

Where:

• p : is to only gather performance data
• d : delay in seconds between statistics snapshots
• i : is the number of iteration 
• w : is to specify a datastore for the output file

Once the command completes, the collected data will be saved in a zipped file stored on the location provided in command

Before running the esxtop command in replay mode, we have to extract the zipped file

Once the file is extracted, you will find a new directory. 

To run the esxtop in replay mode, run esxtop command with -R and name of extracted directory. In my case I had to run

If replay mode does not start and you get an error message similar to “No such file or directory esxtop […] all vm-support snapshots have been used“, then don’t be disheartened. The fix to this problem is “go inside the directory and run reconstruct.sh file.

Now when you run extop -R command, you will be able to see the data. 

Note: The last line in the output: “Read stats from …” indicates that ESXTOP is running in replay mode

                                       Use vScsiStats to gather storage performance data

vscsiStats is a tool controls vscsi data collection for virtual machine disk disk I/O workload characterization. Users can start and stop online data data collection for particular VMs as well as print out online histogram data. 

To use vscsiStats for Storage Performance Analysis, there is an awesome document created by drummonds on VMTN

To see a full list of options that can be used with vscsi command, run vscsiStats -h

We need to have VMWorldID before we can use vscsiStats command to analyze VM disk. To do so run vscsiStats -l command and note the world groupID

To start collecting data: vscsiStats -s -w <WorldGroupID>

To stop collecting data: vscsiStats -x

You can choose one of the following to see the information in histogram:

• all
• ioLength
• seekDistance
• outstandingIOs
• latency
• interarrival

Note: vscsiStat should be running before we can view the info in histogram

To see the collection: vscsiStats -p latency 

You can redirect the screen output to a file by running command: 

You can reset the statistics by typing: vscsiStats -r

                                        Use esxtop / resxtop to collect performance data:

I am not pretty sure what is the goal of this objective. This may or may not include deciphering verious performance related issues via esxtop command or using vm-support command with (-p) option. For general troubleshooting I followed this article by  Matt Callaway

Given esxtop / resxtop output, identify relative performance data for capacity planning purposes

We aready covered this part earlier in this blogpost.

I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Additional Sources

ESXTOP Troubleshooting Poster

Using vscsiStats for Storage Performance Analysis

Interpreting esxtop Statistics