Objective 5.3 of VCAP6-Deploy exam covers following topics:

• Generate vSphere log bundles
• Configure and test centralized logging
• Analyze log entries to obtain configuration information
• Analyze log entries to identify and resolve issues
• Configure logging levels for vSphere

                                                        Generate vSphere log bundles

There are various ways to view/generate log bundles of Esxi host and vCenter server. We will look at all of them one by one. I will start with Esxi host logs first.

1: From the DCUI 

2: Esxi host Web Browser: https://esxi_fqdn_or_ip/host

3: C# client : Connect directly to Esxi host and from home menu click on system logs

From the drop-down menu, select the log and entry you want to view.

4: Web Client : Login to vSphere Web Client and select a vcenter server from inventory and navigate to Monitor > System Logs and click on Export System Logs and select an Esxi host from the list. Optionally you can include vCenter server and web client logs as well alongwith host logs. 

Specify which log files you need to export.

5: Power CLI: Run below command to generate host log via PowerCLI

6: Via Esxi command line: Connect to Esxi user via SSH and login via root user and run command: /usr/bin/vm-support

vCenter Logs

1: Web browser : https://vcenter-FQDN/appliance/support-bundle or https://psc-FQDN/appliance/support-bundle

After you enter credentials, a support bundle will start downloading

2: PowerCLI: Run below command to generate vCenetr server log via PowerCLI

3: Web Client: Login to vSphere Web Client and navigate to Administration > System Configuration > Objects > Nodes. Select a node and from Action dropdown menu select Export Support Bundles

Select what you want to export and whats not and click on “Export Support Bundle”

4: Command Line: Connect to the vCenter Server or PSC appliance via root user over SSH and switch to bash shell by typing “Shell” and run command : vc-support -l

Command will take 5-7 minutes to generate the log bundle. 

Once it is completed, the Support bundle will be saved in the format of “vc-<FQDN_of-PSC>-<Date>.tgz.” under /Storage/log

                                               Configure and test centralized logging

An Esxi host or vCenter server (VCSA) can be configured to forward the system logs to a remote Syslog Server. 

Forwarding Esxi host logs to syslog server:

ESXi 5.0 and higher hosts run a syslog service (vmsyslogd) that provides a standard mechanism for logging messages from the VMkernel and other system components. By default in ESXi, these logs are placed on a local scratch volume or a ramdisk.

To preserve the logs further, ESXi can be configured to place these logs to an alternate storage location on disk and to send the logs across the network to a syslog server. Follow below steps to configure syslog settings for an Esxi host

1: SSH to Esxi host and login via root user and run below command: 

2: Reload the syslog service

3: Open the firewall ports for syslog on Esxi host

4: Test the connectivity to syslog server

There are other ways to achieve the same. I have wrote an article on this topic in past so I am not covering those steps again. 

Forwarding VCSA logs to syslog server:

VCSA comes installed with syslog-ng by default which is used to provide the vSphere Syslog Collector functionality as well as the local syslog client for the VCSA itself.

To forward VCSA logs to a remote syslog server login to vSphere Web Client using administrator@sso-domain and navigate to Administration > System Configuration > Nodes > Select a node from list > Related Objects and search for syslog

Edit the syslog settings using teh pencil button and fill up following details

Common Log Level: The supported values for this field are:

• * : include all log files.
• info: Only informational log files are redirected to the remote machine.
• notice: Only notices are redirected to the remote machine. A notice message indicates a normal but significant condition.
• warn: Only warnings are redirected to the remote machine.
• error: Only error messages are redirected to the remote machine.
• crit: Only critical log files are redirected to the remote machine.
• alert: Only critical log files are redirected to the remote machine.
• emerg: Only emergency log files are redirected to the remote machine. An emergency message indicates that the system has stopped responding and cannot be used.

Remote Syslog Host: The IP address of the host you want to use for storing ESXi messages and logs. This is also the IP address of the remote syslog server on the network you use to redirect logs and ESXi messages

Remote Syslog Port: The port number to use for communication with the machine to which you want to export log files.

Remote Syslog Protocol: The communication protocol that Syslog uses. Available protocols are TCP, UDP, and TLS.

After supplying all the values, click on Restart from Actions menu for changes to take effect. 

For forwarding logs from a Windows based vCenter, William lam has wrote and excellent article. Feel free to check it out.

                            Analyze log entries to obtain configuration information

To check the configuration file browse to the host using a web browser https://host-fqdn-ip/host

The common configuration files that can be viewed from here are: host file, license file, certificate files and host agent config file etc.

Auditing Esxi Shell Logins and Commands

Esxi logs reside in /var/log directory. To review these logs, ssh to an Esxi host directly and login via root user and review following log files

Auth.log : Here you can see login failure/success messages when a user attempt to login to Esxi host

Shell.log: This log file gives you a list of commands that were fired via Esxi shell.

                                    Analyze log entries to identify and resolve issues

Analyzing logs and troubleshooting issues comes with experience and its very hard to write on this topic as every problem is unique in some way and the problems varies from environment to environment as every environment has their own design and stuffs are setup’ed differently.

However VMware has published some very handy KB’s/Docs for how to get started with anaylyzing issues and performing troubleshooting. I have listed few of them:

• SSO and PSC – KB-2113115 
• vCenter Server service – KB-1003926
• vCenter Server connectivity – KB-1029135
• VM resource contention, configuration and operation – KB-1007819
• VMware tools Installation – KB-1003908
• Fault Tolerant network latency – KB-1013428

Esxi log: Esxi logs are present in /var/run/log

vCenter Logs: 

VMware KB-2110014 tells the log location of various log files that resides on vCenter appliance. And this article explains about Esxi Log File Locations.

                                        Configure logging levels for vSphere

Changing Esxi host logging level

Global Log Level: Select an Esxi host and navigate to Manage > Settings > Advanced System Settings and edit the setting Config.HostAgent.log.level

You can choose one of the option as shown in below screenshot

To change logging level of vpxa, edit the parameter Vpx.Vpxa.config.log.level

Changing vCenter server logging level

To change overall logging level of vCenter server, select a VC from inventory and navigate to Manage > Settings > general and click on Edit button.

Choose one of the logging level from the list as shown in below screenshot. 

This is explanation of each logging level

Note: Changes done to the logging level via the vSphere Client or vSphere Web Client do not persist after a reboot and are overwritten by the default values in the vpxd.cfg file. To make permanent log level modifications, you must edit the vpxd.cfg file. VMware KB-1004795 has the steps for modifying the vpxd.cfg file. 

I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing. Be sociable 🙂