For security purposes, it is best practice to store all critical logs to a secured centralized syslog server, and keep the retention rate set for at least 1 year.
In this example, there are 4 servers:
syslog01 (192.168.1.200)
web01-local (192.168.1.201)
web02-local (192.168.1.202)
web03-local (192.168.1.203)
The goal here is to store all the system’s logs, as well as the individual Apache vhost’s logs to the central log server.
Install and perform initial configuration of rsyslog
On syslog01, confirm rsyslog is installed and running by:
[root@syslog01 ~]# yum install rsyslog
[root@syslog01 ~]# chkconfig rsyslog on
[root@syslog01 ~]# service rsyslog start
Open the firewall to allow inbound connections over port 514 to syslog01 by:
[root@syslog01 ~]# vim /etc/sysconfig/iptables
...
-A INPUT -s 192.168.1.0/24 -p tcp -m tcp --dport 514 -j ACCEPT
...
[root@syslog01 ~]# service iptables restart
Finally, create the directory that will be storing the logs:
[root@syslog01 ~]# mkdir -p /var/log/remote
[root@syslog01 ~]# chown root:root /var/log/remote
[root@syslog01 ~]# chmod 700 /var/log/remote
Determine log storage style on central log server
Here is where it needs to be decided how to store the logs on the central logging server. I am unaware of any recommend best practices as there are many ways to go about this. I’ll outline 4 common ways below:
– Option 1: Store each server’s logs in its own directory.
– Option 2: Store each server’s logs in its own directory with natural log rotation
– Option 3: Store each server’s logs in its own directory, broken down by program name
– Option 4: Store all server’s logs in a single file
Option 1: Store each server’s logs in its own directory
This will configure rsyslog to store the servers logs in a directory named after the remote server’s hostname. Log rotation will happen daily via logrotate.
To implement this, first create a backup of the existing /etc/rsyslog.conf, and create a new one as shown below. The parts I added or modified are in bold for reference:
[root@syslog01 ~]# mv /etc/rsyslog.conf /etc/rsyslog.orig
[root@syslog01 ~]# vim /etc/rsyslog.conf
# rsyslog v5 configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
$ModLoad imudp
# Moved below for RuleSets
#$UDPServerRun 514
# Provides TCP syslog reception
$ModLoad imtcp
# Moved below for RuleSets
#$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
$template RemoteHost, "/var/log/remote/%HOSTNAME%/syslog.log"
#### RULES ####
$RuleSet local
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
local0.* /var/log/local0.log
local1.* /var/log/local1.log
local2.* /var/log/local2.log
local3.* /var/log/local3.log
local4.* /var/log/local4.log
local5.* /var/log/local5.log
local6.* /var/log/local6.log
# Log all messages also to the central log directory
*.* ?RemoteHost
# Bind the above ruleset as default
$DefaultRuleset local
# This uses the template above
$RuleSet remote
*.* ?RemoteHost
# Bind the above ruleset for remote logs
$InputUDPServerBindRuleset remote
$UDPServerRun 514
$InputTCPServerBindRuleset remote
$InputTCPServerRun 514
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###
Then restart rsyslog:
[root@syslog01 ~]# service rsyslog restart
Now setup logrotate to rotate out the logs, compress them, and keep them on file for 1 year as shown below:
[root@syslog01 ~]# vim /etc/logrotate.d/centralrsyslog
/var/log/remote/*/*.log
{
daily
rotate 365
missingok
daily
compress
delaycompress
sharedscripts
postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
endscript
}
Finally, test the log rotation by running:
[root@syslog01 ~]# logrotate -f /etc/logrotate.d/centralrsyslog
Option 2: Store each server’s logs in its own directory with natural log rotation
Both the local and remote system logs will be stored in its own directory as follows:
/var/log/remote/2015-12_syslog01/syslog.conf
/var/log/remote/2015-12_web01-local/syslog.conf
/var/log/remote/2015-12_web02-local/syslog.conf
/var/log/remote/2015-12_web03-local/syslog.conf
As the months change, syslog will automatically create a new folder accordingly for that month, creating a type of natural log rotation.
To implement this, first create a backup of the existing /etc/rsyslog.conf, and create a new one as shown below. The parts I added or modified are in bold for reference:
[root@syslog01 ~]# mv /etc/rsyslog.conf /etc/rsyslog.orig
[root@syslog01 ~]# vim /etc/rsyslog.conf
# rsyslog v5 configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
$ModLoad imudp
# Moved below for RuleSets
#$UDPServerRun 514
# Provides TCP syslog reception
$ModLoad imtcp
# Moved below for RuleSets
#$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
$template RemoteHost, "/var/log/remote/%$YEAR%-%$MONTH%_%HOSTNAME%/syslog.log"
#### RULES ####
$RuleSet local
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
local0.* /var/log/local0.log
local1.* /var/log/local1.log
local2.* /var/log/local2.log
local3.* /var/log/local3.log
local4.* /var/log/local4.log
local5.* /var/log/local5.log
local6.* /var/log/local6.log
# Log all messages also to the central log directory
*.* ?RemoteHost
# Bind the above ruleset as default
$DefaultRuleset local
# This uses the template above
$RuleSet remote
*.* ?RemoteHost
# Bind the above ruleset for remote logs
$InputUDPServerBindRuleset remote
$UDPServerRun 514
$InputTCPServerBindRuleset remote
$InputTCPServerRun 514
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###
Then restart rsyslog:
[root@syslog01 ~]# service rsyslog restart
Option 3: Store each server’s logs in its own directory, broken down by program name
This one will break down the logs to store them in a directory named after the remote server’s hostname, and further more break them down by program name.
To implement this, first create a backup of the existing /etc/rsyslog.conf, and create a new one as shown below. The parts I added or modified are in bold for reference:
[root@syslog01 ~]# mv /etc/rsyslog.conf /etc/rsyslog.orig
[root@syslog01 ~]# vim /etc/rsyslog.conf
# rsyslog v5 configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
$ModLoad imudp
# Moved below for RuleSets
#$UDPServerRun 514
# Provides TCP syslog reception
$ModLoad imtcp
# Moved below for RuleSets
#$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
$template RemoteHost, "/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
#### RULES ####
$RuleSet local
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
local0.* /var/log/local0.log
local1.* /var/log/local1.log
local2.* /var/log/local2.log
local3.* /var/log/local3.log
local4.* /var/log/local4.log
local5.* /var/log/local5.log
local6.* /var/log/local6.log
# Log all messages also to the central log directory
*.* ?RemoteHost
# Bind the above ruleset as default
$DefaultRuleset local
# This uses the template above
$RuleSet remote
*.* ?RemoteHost
# Bind the above ruleset for remote logs
$InputUDPServerBindRuleset remote
$UDPServerRun 514
$InputTCPServerBindRuleset remote
$InputTCPServerRun 514
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###
Then restart rsyslog:
[root@syslog01 ~]# service rsyslog restart
Now setup logrotate to rotate out the logs, compress them, and keep them on file for 1 year as shown below:
[root@syslog01 ~]# vim /etc/logrotate.d/centralrsyslog
/var/log/remote/*/*.log
{
daily
rotate 365
missingok
daily
compress
delaycompress
sharedscripts
postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
endscript
}
Finally, test the log rotation by running:
[root@syslog01 ~]# logrotate -f /etc/logrotate.d/centralrsyslog
Option 4: Store all server’s logs in a single file
This is useful is you want to be able to use grep to find the information you need from a single file. Just keep in mind that if you have 10-20+ servers, this log file could get very large, even with a daily log rotation in place.
This will tell rsyslog to store ALL your servers logs in /var/log/remote/syslog.log.
To implement this, first create a backup of the existing /etc/rsyslog.conf, and create a new one as shown below. The parts I added or modified are in bold for reference:
[root@syslog01 ~]# mv /etc/rsyslog.conf /etc/rsyslog.orig
[root@syslog01 ~]# vim /etc/rsyslog.conf
# rsyslog v5 configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
$ModLoad imudp
# Moved below for RuleSets
#$UDPServerRun 514
# Provides TCP syslog reception
$ModLoad imtcp
# Moved below for RuleSets
#$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
$template RemoteHost, "/var/log/remote/syslog.log"
#### RULES ####
$RuleSet local
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
local0.* /var/log/local0.log
local1.* /var/log/local1.log
local2.* /var/log/local2.log
local3.* /var/log/local3.log
local4.* /var/log/local4.log
local5.* /var/log/local5.log
local6.* /var/log/local6.log
# Log all messages also to the central log directory
*.* ?RemoteHost
# Bind the above ruleset as default
$DefaultRuleset local
# This uses the template above
$RuleSet remote
*.* ?RemoteHost
# Bind the above ruleset for remote logs
$InputUDPServerBindRuleset remote
$UDPServerRun 514
$InputTCPServerBindRuleset remote
$InputTCPServerRun 514
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###
Then restart rsyslog:
[root@syslog01 ~]# service rsyslog restart
Now setup logrotate to rotate out the logs, compress them, and keep them on file for 1 year as shown below:
[root@syslog01 ~]# vim /etc/logrotate.d/centralrsyslog
/var/log/remote/*/*.log
{
daily
rotate 365
missingok
daily
compress
delaycompress
sharedscripts
postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
endscript
}
Finally, test the log rotation by running:
[root@syslog01 ~]# logrotate -f /etc/logrotate.d/centralrsyslog
Setup clients to forward their logs to your central log server
For this, I am opting to use TCP. Append the following at the bottom of /etc/rsyslog.conf on each client server:
[root@syslog01 ~]# vim /etc/rsyslog.conf
*.* @@192.168.1.200:514
Then restart Rsyslog:
[root@syslog01 ~]# service rsyslog restart
Apache logs
If there was a requirement within the organization to also send Apache error logs access logs to the central syslog server, that can be done pretty easily. However as these can get very large, make sure you have enough disk space on your central syslog server to handle this.
Apache does not have the ability to natively send over the logs, so logger must be used. The following would need to be done on each vhost configured for both 80 and 443. Be sure to change the tags for each site accordingly in the logger entry below:
[root@syslog01 ~]# vim /etc/httpd/vhost.d/example.com.conf
...
CustomLog "|/usr/bin/logger -p local0.info -t example.com-access" combined
ErrorLog "|/usr/bin/logger -p local0.error -t example.com-error"
...
Restart Apache when done
[root@syslog01 ~]# service httpd restart
Additional RemoteHost templates
As seen in the examples above, there are a number of ways you can work with the directories created. Some common ones that I know of are below:
Each folder is labeled with the hostname it came from:
$template RemoteHost, "/var/log/remote/%HOSTNAME%/syslog.log"
Each folder is labeled with the hostname and IP address it came from:
$template RemoteHost, "/var/log/remote/%HOSTNAME%-%fromhost-ip%/syslog.log"
Each folder is labeled with the hostname, and break down the logs within by program name:
$template RemoteHost, "/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
Each folder is labeled with the year, month and hostname, creating a type of natural log rotation:
$template RemoteHost, "/var/log/remote/%$YEAR%-%$MONTH%_%HOSTNAME%/syslog.log"