Category: Mail

Centralized mail relay server on CentOS 7

Lets say you have dozens or hundreds of servers that all need to send mail out directly to the internet. This becomes a headache as you need to open up your firewall to allow all these servers outbound access over port 25 and your mail logs are scattered among all those servers.

Having a centralized mail relay server solves for this by serving as a central location for mail logs and only opening the firewall for one server to allow outbound port 25 access. All the other servers simply send their mail to this central mail relay server to handle sending mail, which alleviates the need for unnecessary outbound access for those other nodes.

This guide will discuss how to setup a centralized mail relay server for the sole purpose of sending only outbound email. The servers used in this guide as an example will be:

smtp-relay001.example.com - 192.168.1.100
web01.example.com - 192.168.1.101
web02.example.com - 192.168.1.102

There are some basic prerequisites that must be meet before beginning to help ensure successful email delivery:

  • The hostname of the relay server must be a FQDN, ie: smtp-relay001.example.com
  • There must be a corresponding A record setup in DNS that matches the hostname
  • There must be a corresponding PTR record (reverse DNS) setup in that matches the hostname
  • Setup an SPF record in DNS for your central mail relay server
  • Ensure your relay server is configured to ONLY accept mail from your private network to prevent it from becoming an open relay!

To reiterate the last point, ensure that your central mail relay server ONLY accepts mail from your private network. Opening it up to the world makes you an open relay which will get you blacklisted quickly. Use a dedicated firewall to block inbound 25 and 587 access to the relay server for added protection against a configuration error.

Setup central mail relay server (smtp-relay001.example.com)

First, confirm your hostname is setup properly:

[root@smtp-relay001 ~]# vim /etc/hosts
...
192.168.1.100 smtp-relay001.example.com smtp-relay001
...

[root@smtp-relay001 ~]# hostnamectl set-hostname smtp-relay001.example.com
[root@smtp-relay001 ~]# hostname smtp-relay001.example.com
[root@smtp-relay001 ~]# systemctl restart rsyslog

Now install postfix if it is not already installed:

[root@smtp-relay001 ~]# yum install postfix
[root@smtp-relay001 ~]# systemctl enable postfix

Set postfix to listen on your private IP address and only answer to servers within your network, which in my case is the 192.168.1.0/24 network:

[root@smtp-relay001 ~]# vim /etc/postfix/main.cf
...
inet_interfaces = 192.168.1.100
mydestination = $myhostname, localhost.$mydomain, $mydomain
mynetworks = 192.168.1.0/24, 127.0.0.0/8
...

Then setup a SSL certificate for use with TLS:

[root@smtp-relay001 ~]# openssl genrsa -out /etc/postfix/server.key 2048
[root@smtp-relay001 ~]# openssl req -new -x509 -key /etc/postfix/server.key -out /etc/postfix/server.crt -days 3650
[root@smtp-relay001 ~]# chmod 600 /etc/postfix/server.key

Add the following TLS configuration to the bottom of the postfix configuration:

[root@smtp-relay001 ~]# vim /etc/postfix/main.cf
...
# Enable TLS
smtpd_use_tls = yes
smtpd_tls_key_file = /etc/postfix/server.key
smtpd_tls_cert_file = /etc/postfix/server.crt
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes
smtpd_tls_session_cache_timeout = 3600s
tls_random_source = dev:/dev/urandom
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1

Set the server to accept TLS connections by:

[root@smtp-relay001 ~]# vim /etc/postfix/master.cf
...
submission inet n       -       n       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
...

Confirm postfix syntax looks good

[root@smtp-relay001 ~]# postfix check

Now restart Postfix to apply the changes:

[root@smtp-relay001 ~]# systemctl restart postfix

Finally, open up the software firewall (or the dedicated firewall) to allow inbound 25 and 587 requests from other servers within your private network by:

# Firewalld
[root@smtp-relay001 ~]# firewall-cmd --permanent --new-zone=postfix
[root@smtp-relay001 ~]# firewall-cmd --permanent --zone=postfix --add-port=25/tcp
[root@smtp-relay001 ~]# firewall-cmd --permanent --zone=postfix --add-port=587/tcp
[root@smtp-relay001 ~]# firewall-cmd --permanent --zone=postfix --add-source=192.168.1.0/24
[root@smtp-relay001 ~]# firewall-cmd --reload

# iptables
[root@smtp-relay001 ~]# vim /etc/sysconfig/iptables
...
-A INPUT -p tcp -m tcp --dport 25 -s 192.168.1.0/24 -m comment --comment "postfix" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 587 -s 192.168.1.0/24 -m comment --comment "postfix" -j ACCEPT
...
[root@smtp-relay001 ~]# service iptables restart

Setup client servers running postfix to relay through smtp-relay001

First, confirm postfix is installed:

[root@web01 ~]# yum install postfix
[root@web01 ~]# systemctl enable postfix

Configure postfix to relay mail to smtp-relay001, only accept mail from localhost, and configure the relay host:

[root@web01 ~]# vim /etc/postfix/main.cf
...
inet_interfaces = loopback-only
mydestination= # leave blank
myhostname = ENTER_SERVER_HOSTNAME_HERE
mynetworks=127.0.0.0/8 [::1]/128
myorigin = $myhostname
relayhost = 192.168.1.100
local_transport=error: local delivery disabled
...

Confirm postfix syntax looks good:

[root@web01 ~]# postfix check

Restart postfix to apply the changes:

[root@web01 ~]# systemctl restart postfix

Confirm email can send outbound by sending a message, then checking the mail logs to ensure you see it relay through the relay server:

[root@web01 ~]# yum install mailx
[root@web01 ~]# echo "Testing" | mail -s "Test from web01" [email protected]
[root@web01 ~]# tail -f /var/log/maillog

Setup client servers running sendmail to relay through smtp-relay001

While I rarely run across sendmail nowadays, there are still some servers that are using it. If one of your servers is running sendmail, you can set the relay host by replacing DS with DS192.168.1.100 in your sendmail configuration as shown below:

[root@web01 ~]# vim /etc/mail/sendmail.cf
...
DS192.168.1.100
...
[root@web01 ~]# service sendmail restart

Confirm email can send outbound by sending a message, then checking the mail logs to ensure you see it relay through the relay server:

[root@web01 ~]# yum install mailx
[root@web01 ~]# echo "Testing" | mail -s "Test from web01" [email protected]
[root@web01 ~]# tail -f /var/log/maillog  # or /var/log/mail.log

Postfix – Flush mail queue

I have seen servers where the Postfix mail queue is jammed up with mail, perhaps from a programming error on the application, or maybe spam if they were web hacked. During these times, you may just want to purge the queue since you don’t want the messages going out. Below are some very simple methods of doing this:

How to remove all mail in the postfix queue:

[root@web01 ~]# postsuper -d ALL

How to remove all email in the deferred queue

[root@web01 ~]# postsuper -d ALL deferred

Remove all email in the queue for a specific domain:

[root@web01 ~]# postqueue -p | tail -n +2 | awk 'BEGIN { RS = "" } /@example\.com/ { print $1 }' | tr -d '*!' | postsuper -d -

Remove all email in the queue from a specific email address: 

[root@web01 ~]# postqueue -p | tail -n +2 | awk 'BEGIN { RS = "" } /user@example\.com/ { print $1 }' | tr -d '*!' | postsuper -d -

How Qmail Works

Qmail is a very compartmentalized program. Its broken down into multiple tiny programs that govern a very specific piece of the MTA process. This guide documents how Qmail handles email in a nutshell.

Below is a rough diagram of what Qmail looks like:
Qmail Diagram

Messages can enter the mail server in one of 2 ways, either the message came from a remote mailserver like hotmail.com, or the message is being sent from the local server (ie. imap, webmail, mail functions, etc) The 2 daemons that are responsible for this are:

1. qmail-smtpd –> handles mail coming from an outside mail server. ie. hotmail.com

2. qmail-inject –> handles any messages generated locally by the server. ie. imap, webmail, or php mail functions, etc. This service injects the messages directly into the mail queue.

The primary objective of qmail-smtpd and qmail-inject is to pass the message along to qmail-queue.

3. qmail-queue –> This is a complicated program. This writes all the messages to the central queue directory: /var/qmail/queue/. The qmail-queue program can be invoked by qmail-inject for locally generated messages, qmail-smtpd for messages received through SMTP, qmail-local for forwarded messages, or qmail-send for bounced messages. If this is confusing, just remember that this is the program that acually writes the messages to the mail queue. Now, if you are curious like me and want to know the nitty gritty, here it is. /var/qmail/queue is comprised of 5 directories. pid/, mess/, intd/, todo/, info/, and remote/.

Below is a diagram that shows how the message gets handled by qmail-queue during the various message “stages”. Next to each folder I also noted which program controls the message at that particular point in time.

pid/111 --  (S1)  # qmail-queue
          \_ mess/111 (S2)  # qmail-queue
                          |
                          |
                      _ intd/111 (S3)  # qmail-queue
                     /
          todo/111 -- (S4)  # qmail-queue
              |
              |
          info/111 -- local/111  (S4 - S5)  # qmail-send
              |
              |
          remote/111 (S4 - S5)  # qmail-send

Key:
# qmail-send --> responsible for this part of queue
# qmail-queue --> responsible for this part of queue
S1 -->  -mess -intd -todo -info -local -remote -bounce
S2 --> +mess -intd -todo -info -local -remote -bounce
S3 --> +mess +intd -todo -info -local -remote -bounce
S4 --> +mess ?intd +todo ?info ?local ?remote -bounce (queued)
S5 --> +mess -intd -todo +info ?local ?remote ?bounce (preprocessed)

Here are all possible states for a message. + means a file exists; - means it does not exist; ? means it may or may not exist in that folder

It is also well documented in the qmail src file called INTERNALS which explains it better than I can!

Short and sweet overview of qmail-queue: It is responsible for writing the message to the queue.

4. qmail-send –> This takes the message from qmail-queue and passes the message either to qmail-rspawn (for remote delivery) or it sends the message to qmail-lspawn (for local delivery)

5a. qmail-rspawn (remote delivery) –> This sends the message to the remote mail server (ie. yahoo.com)
– qmail-remote –> This transmits the message to the remote mail server.

5b. qmail-lspawn (local delivery) –> This sends the message for local delivery.
– qmail-local –> This passes the message off to a local delivery agent. It reads the users .qmail-default first, which basically just tells qmail-local that vdelivermail is going to handle delivery.

– vdelivermail -> This delivers the mail to the local users. It locates the users Maildir and passes the message off to preline. (preline passes teh mail to other filters or commands) In the users maildir, search for a .qmail file that relates to the user and cat the file. If a cat .qmail-USERNAME doesn’t exist, then it defaults to .qmail-default, which tells it to send the message to procmail.

– procmail -> Procmail performs the mail filtering and local delivery to the mailbox. In procmail, this is where you can send the message to spamassassin or another filtering agent for processing before delivery.

– spamassassin -> Spam filtering. Take special note of the spamassassin versions and permissions, and the spamd vs spamc methods.

Below are qmail’s configuration files, found within /var/qmail/control:

Control file                    Purpose

badmailfrom             blacklisted From addresses
bouncefrom              username of bounce sender
bouncehost              hostname of bounce sender
concurrencyincoming     max simultaneous incoming SMTP connections
concurrencylocal        max simultaneous local deliveries
concurrencyremote       max simultaneous remote deliveries
defaultdomain           domain name
defaulthost             host name
databytes               max number of bytes in message (0=no limit)
doublebouncehost        host name of double bounce sender
doublebounceto          user to receive double bounces
locals                  domains that we deliver locally
morercpthosts           secondary rcpthosts database
queuelifetime           seconds a message can remain in queue
rcpthosts               domains that we accept mail for
smtproutes              artificial SMTP routes
timeoutconnect          how long, in seconds, to wait for SMTP connection
timeoutremote           how long, in seconds, to wait for remote server
timeoutsmtpd            how long, in seconds, to wait for SMTP client
virtualdomains          virtual domains and users

How to read the Qmail logs

This is a quick guide on how to read the Qmail logs.

All message activity is written to /var/log/qmail/current. There can be a lot of information here, so lets break it down line by line. Below is a snippet from /var/log/qmail/current. I added numbers on the left hand side for the sake of learning.

1.  @40000000461d81581ec60f34 new msg 5915497
2.  @40000000461d81581eda8194 info msg 5915497: bytes 22122 from <[email protected]> qp 99024 uid 89
3.  @40000000461d8158214e737c starting delivery 4088258: msg 5915497 to local myuser@localhost
4.  @40000000461d81582155ca64 status: local 2/10 remote 0/60
5.  @40000000461d815824de127c delivery 4088258: success: did_1+0+0/
6.  @40000000461d815824f572dc end msg 5915497

Holy @#$%!, what does all this mean? Here it is, line by line:

1. This indicates that a new message has entered the queue. It is denoted by number: 5915497
2. This tells us where the message was from. In this case: [email protected]
3. Here, we see that the message is trying to deliver to a local user: myuser@localhost. Note the delivery sub id number: 4088258
4. This tells us what the queue volume is like. Not important at this moment.
5. This lets us know the message was delivered successfully to myuser@localhost. Note again the delivery sub id number: 4088258
6. Now qmail says, okay, the message denoted by the number: 5915497 is complete.

So when looking at the logs, first locate a from address or destination address in the logs. Once that is found, find the message id number (should be up 1 or 2 lines) and once you find that, you can discover what the message is doing when in the queue.