In this post we will learn how to configure Esxi-6 hosts to send the logs to a centralized syslog server.
Purpose of configuring syslog server?
As per VMware KB-2003322
ESXi 5.0 and higher hosts run a syslog service (vmsyslogd) that provides a standard mechanism for logging messages from the VMkernel and other system components. By default in ESXi, these logs are placed on a local scratch volume or a ramdisk.
To preserve the logs further, ESXi can be configured to place these logs to an alternate storage location on disk and to send the logs across the network to a syslog server.
Retention, rotation, and splitting of logs received and managed by a syslog server are fully controlled by that syslog server. ESXi cannot configure or control log management on a remote syslog server.
How to configure Esxi hosts for centralized logging?
There are various ways to configure syslog settings on Esxi hosts. These includes:
1: Using esxcli command on Esxi host.
2: Using vSphere Web-Client.
3: Using vSphere Thick client.
4: Using PowerCli.
5: Using Host Profiles.
We will look individually on all available method one by one. Let’s get started.
Before configuring esxi hosts to send logs to syslog server, we need to have a syslog server in our environment. I have configured my syslog server on a CentOS 6 box following instructions illustrated here
I added additional 2 lines at the bottom of rsyslog.conf file so that all hosts should have their logs in their individual folder
|
1 2 |
$template TmplAuth, "/var/log/%HOSTNAME%/%PROGRAMNAME%.log" *.* ?TmplAuth |
Configuring Syslog Using esxcli utility
The command to configure syslog settings on Esxi hosts is esxcli system syslog config
Lets first see what are the available options with this command.
[root@esxi01:~] esxcli system syslog config set –help
With this command we have following options available:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
--check-ssl-certs : Verify remote SSL certificates against the local CA Store --default-rotate : Default number of rotated local logs to keep --default-size : Default size of local logs before rotation, in KiB --default-timeout : Default network retry timeout in seconds if a remote server fails to respond --drop-log-rotate : Number of rotated dropped log files to keep --drop-log-size : Size of dropped log file before rotation, in KiB --logdir : The directory to output local logs to --logdir-unique : Place logs in a unique subdirectory of logdir, based on hostname --loghost : The remote host(s) to send logs to --queue-drop-mark : Message queue capacity after which messages are dropped --reset : Reset values to default |
Next is to retrieve the current syslog configuration.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
[root@esxi01:~] esxcli system syslog config get Default Network Retry Timeout: 180 Dropped Log File Rotation Size: 100 Dropped Log File Rotations: 10 Enforce SSLCertificates: false Local Log Output: /scratch/log Local Log Output Is Configured: false Local Log Output Is Persistent: true Local Logging Default Rotation Size: 1024 Local Logging Default Rotations: 8 Log To Unique Subdirectory: false Message Queue Drop Mark: 90 Remote Host: <none> |
As you can see Remote Host is currently set to none, which means there is no syslog server configured on Esxi host.
To set syslog server on host, we can use the below command
[root@esxi01:~] esxcli system syslog config set –loghost syslogsrv.alex.local
Next is to reload the syslog configuration
[root@esxi01:~] esxcli system syslog reload
Open the firewall ports for syslog on Esxi host
[root@esxi01:~] esxcli network firewall ruleset set -r syslog -e true
Test the connectivity to syslog server
[root@esxi01:~] nc -z syslogsrv.alex.local 514
Connection to syslogsrv.alex.local 514 port [tcp/shell] succeeded!
And that’s it. On checking /var/log/messages on my syslog server, I was able to see logs getting populated
|
1 2 3 4 5 |
Aug 4 15:54:03 esxi01 Vpxa: info vpxa[50A90B70] [Originator@6876 sub=vpxLro opID=PollQuickStatsLoop-77188b05-47] [VpxLRO] -- BEGIN task-internal-134340 -- vpxa -- vpxapi.VpxaService.fetchQuickStats -- 52a22f84-3c8d-eee9-f2a5-dfa16fdc22ac Aug 4 15:54:03 esxi01 Vpxa: info vpxa[50A90B70] [Originator@6876 sub=vpxLro opID=PollQuickStatsLoop-77188b05-47] [VpxLRO] -- FINISH task-internal-134340 Aug 4 15:54:14 esxi01 Hostd: info hostd[37F40B70] [Originator@6876 sub=Libs] SOCKET creating new socket, connecting to /var/run/vmware/usbarbitrator-socket Aug 4 15:54:14 esxi01 Hostd: info hostd[37F40B70] [Originator@6876 sub=Libs] SOCKET connect failed, error 2: No such file or directory Aug 4 15:54:14 esxi01 Hostd: warning hostd[39540B70] [Originator@6876 sub=UserDirectory] Group lookup failed for 'ALEXESX Admins' |
Also i verified that under/var/log i have a folder for esxi01 host and it has all the log files from my host
|
1 2 3 4 |
[root@syslogsrv esxi01]# pwd /var/log/esxi01 [root@syslogsrv esxi01]# ls crond.log Fdm.log Hostd.log hostd-probe.log lwsmd.log mark.log Rhttpproxy.log shell.log smartd.log syslog.log vmkernel.log vobd.log Vpxa.log |
Configuring syslog settings via Web-Client
To configure syslog on Esxi host via Web-Client, navigate to Host and Cluster and select Esxi host > Manage > Settings > Advanced System Settings and in search box type syslog.global and verify value for Syslog.global.loghost
Also you can define following parameters:
- Syslog.global.logDirUnique: True/False. Set to true will create individual directories per host (w/ host name) in the specified folder.
- Syslog.global.defaultRotate: Max number of logs to keep locally. Does not effect the remote log server retention.
- Syslog.global.defaultSize: Max size (in KB) of each log file before it is rotated or rolled over.
If the value is empty, Edit the settings by clicking on the pencil icon and add the entry as shown below:
Open firewall posts for syslog server by Navigating to Security Profile and click on Edit button
Look for “syslog” rule and if the check box is not selected then select it and click “OK”
Under outgoing connections, verify syslog is now listing. If not then refresh web-client to see entry appears there or not.
Under Services, verify syslog server setting is listed as Running
Also verify syslog service startup policy is set to “Start and stop with host” if you want service to be persistent across host reboot.
Configuring Syslog via vSphere Thick Client
Select Esxi host and go to Configuration tab and select Advanced Settings. Locate Syslog and expand the settings and under global configuration verify if there is any entry for Syslog.global.logHost.
If there are no settings present, enter a new one by adding udp://syslog-IP:514 and hit OK.
Under firewall settings, select the syslog service and hit OK.
Under Service Properties, verify Syslog Server is Running. if not select the service and click on Options button and set service policy to “Start and Stop with host” and start service.
Configuring syslog on Esxi hosts using PowerCli
1: Set the syslog server on all Esxi host using command
|
1 |
get-vmhost| Set-VMHostAdvancedConfiguration -NameValue @{'Config.HostAgent.log.level'='info';'Vpx.Vpxa.config.log.level'='info';'Syslog.global.logHost'='udp://syslogsrv.alex.local:514'} |
2: Open the firewall ports
|
1 |
get-vmhost| Get-VMHostFirewallException |?{$_.Name -eq 'syslog'} | Set-VMHostFirewallException -Enabled:$true |
3: Restart syslog service on all hosts
|
1 2 3 |
$ESXhost=get-vmhost Get-VMHost -name $ESXhost | Get-VMHostService | where {$_.key -eq 'vmsyslogd'} | Start-VMHostService -Confirm:$start |
4: Confirm Esxi hosts syslog settings
|
1 |
get-vmhost | Get-AdvancedSetting -Name Syslog.global.logHost |
Setting syslog via Host profile
Edit your Host Profile and expand Advanced Configuration Settings profile and locate Syslog.global.logHost setting. This is by default set to “Set the option to its default value”
Change the option to “Configure a fixed option” and provide the value of the option in formal protocol://syslogsrv-ip:port. Typically this is udp://syslogsrv-ip:514
Expand Security and Services profile and locate syslog. Checkmark the boxes under Ruleset.
Expand Service Configuration profile and add a sub-profile to it. Set service name as “vmsyslogd” and set service configuration to start and stop with host.
Save the host profile and navigate to Host and Cluster, select the cluster and check Esxi hosts for compliance. You will see your hosts as non-compliant if syslog server is not configured on hosts.
As expected one of my host was non-complaint with profile and was complaining about syslog settings not present on host.
I placed the host in MM and remediated it and things were back to green. Hosts were happy,cluster was also happy and compliant.
And that’s it. We have learnt 5 different ways in which hosts can be configured to send logs to a centralized syslog server. Happy syslogging!!!!
I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing. Be sociable 🙂