People looking to create a load balanced web server solution often ask, how can they keep their web servers in sync with each other? There are many ways to go about this: NFS, lsync, rsync, etc. This guide will discuss a technique using rsync that runs from a cron job every 10 minutes.
There will be two different options presented, pulling the updates from the master web server, and pushing the updates from the master web server down to the slave web servers.
Our example will consist of the following servers:
web01.example.com (192.168.1.1) # Master Web Server
web02.example.com (192.168.1.2) # Slave Web Server
web03.example.com (192.168.1.3) # Slave Web Server
Our master web server is going to the single point of truth for the web content of our domain. Therefore, the web developers will only be modifying content from the master web server, and will let rsync handle keeping all the slave nodes in sync with each other.
There are a few prerequisites that must be in place:
1. Confirm that rsync is installed.
2. If pulling updates from the master web server, all slave servers must be able to SSH to the master server using a SSH key with no pass phrase.
3. If pushing updates from the master down to the slave servers, the master server must be able to SSH to the slave web servers using a SSH key with no passphrase.
To be proactive about monitoring the status of the rsync job, both scripts posted below allow you to perform a http content check against a status file to see if the string “SUCCESS” exists. If something other then SUCCESS is found, that means that the rsync script may have failed and should be investigated. An example of this URL to monitor would be is: 192.168.1.1/datasync.status
Please note that the assumption is being made that your web server will serve files that are placed in /var/www/html/. If not, please update the $status variable accordingly.
Using rsync to pull changes from the master web server:
This is especially useful if you are in a cloud environment and scale your environment by snapshotting an existing slave web server to provision a new one. When the new slave web server comes online, and assuming it already has the SSH key in place, it will automatically grab the latest content from the master server with no interaction needed by yourself except to test, then enable in your load balancer.
The disadvantage with using the pull method for your rsync updates comes into play when you have multiple slave web servers all running the rsync job at the same time. This can put a strain on the master web servers CPU, which can cause performance degradation. However if you have under 10 servers, or if your site does not have a lot of content, then the pull method should work fine.
Below will show the procedure for setting this up:
1. Create SSH keys on each slave web server:
ssh-keygen -t dsa
2. Now copy the public key generated on the slave web server (/root/.ssh/id_dsa.pub) and append it to the master web servers, /root/.ssh/authorized_keys2 file.
3. Test ssh’ing in as root from the slave web server to the master web server
# On web02
ssh [email protected]
4. Assuming you were able to log in to the master web server cleanly, then its time to create the rsync script on each slave web server. Please note that I am assuming your sites documentroot’s are stored in /var/www/vhosts. If not, please change the script accordingly and test!
mkdir -p /opt/scripts/
vi /opt/scripts/pull-datasync.sh
#!/bin/bash
# pull-datasync.sh : Pull site updates down from master to front end web servers via rsync
status="/var/www/html/datasync.status"
if [ -d /tmp/.rsync.lock ]; then
echo "FAILURE : rsync lock exists : Perhaps there is a lot of new data to pull from the master server. Will retry shortly" > $status
exit 1
fi
/bin/mkdir /tmp/.rsync.lock
if [ $? = "1" ]; then
echo "FAILURE : can not create lock" > $status
exit 1
else
echo "SUCCESS : created lock" > $status
fi
echo "===== Beginning rsync ====="
nice -n 20 /usr/bin/rsync -axvz --delete -e ssh [email protected]:/var/www/vhosts/ /var/www/vhosts/
if [ $? = "1" ]; then
echo "FAILURE : rsync failed. Please refer to solution documentation" > $status
exit 1
fi
echo "===== Completed rsync ====="
/bin/rm -rf /tmp/.rsync.lock
echo "SUCCESS : rsync completed successfully" > $status
Be sure to set executable permissions on this script so cron can run it:
chmod 755 /opt/scripts/pull-datasync.sh
Using rsync to push changes from the master web server down to slave web servers:
Using rsync to push changes from the master down to the slaves also has some important advantages. First off, the slave web servers will not have SSH access to the master server. This could become critical if one of the slave servers is ever compromised and try’s to gain access to the master web server. The next advantage is the push method does not cause a serious CPU strain cause the master will run rsync against the slave servers, one at a time.
The disadvantage here would be if you have a lot of web servers syncing content that changes often. Its possible that your updates will not be pushed down to the web servers as quickly as expected since the master server is syncing the servers one at a time. So be sure to test this out to see if the results work for your solution. Also if you are cloning your servers to create additional web servers, you will need to update the rsync configuration accordingly to include the new node.
Below will show the procedure for setting this up:
1. To make administration easier, its recommended to setup your /etc/hosts file on the master web server to include a list of all the servers hostnames and internal IP’s.
vi /etc/hosts
192.168.1.1 web01 web01.example.com
192.168.1.2 web02 web02.example.com
192.168.1.3 web03 web03.example.com
2. Create SSH keys on the master web server:
ssh-keygen -t dsa
3. Now copy the public key generated on the master web server (/root/.ssh/id_dsa.pub) and append it to the slave web servers, /root/.ssh/authorized_keys2 file.
4. Test ssh’ing in as root from the master web server to each slave web server
# On web01
ssh root@web02
5. Assuming you were able to log in to the slave web servers cleanly, then its time to create the rsync script on the master web server. Please note that I am assuming your sites documentroot’s are stored in /var/www/vhosts. If not, please change the script accordingly and test!
mkdir -p /opt/scripts/
vi /opt/scripts/push-datasync.sh
#!/bin/bash
# push-datasync.sh - Push site updates from master server to front end web servers via rsync
webservers=(web01 web02 web03 web04 web05)
status="/var/www/html/datasync.status"
if [ -d /tmp/.rsync.lock ]; then
echo "FAILURE : rsync lock exists : Perhaps there is a lot of new data to push to front end web servers. Will retry soon." > $status
exit 1
fi
/bin/mkdir /tmp/.rsync.lock
if [ $? = "1" ]; then
echo "FAILURE : can not create lock" > $status
exit 1
else
echo "SUCCESS : created lock" > $status
fi
for i in ${webservers[@]}; do
echo "===== Beginning rsync of $i ====="
nice -n 20 /usr/bin/rsync -avzx --delete -e ssh /var/www/vhosts/ root@$i:/var/www/vhosts/
if [ $? = "1" ]; then
echo "FAILURE : rsync failed. Please refer to the solution documentation " > $status
exit 1
fi
echo "===== Completed rsync of $i =====";
done
/bin/rm -rf /tmp/.rsync.lock
echo "SUCCESS : rsync completed successfully" > $status
Be sure to set executable permissions on this script so cron can run it:
chmod 755 /opt/scripts/push-datasync.sh
Now that you have the script in place and tested, its now time to set this up to run automatically via cron. For the example here, I am setting up cron to run this script every 10 minutes.
If using the push method, put the following into the master web servers crontab:
crontab -e
# Datasync script
*/10 * * * * /opt/scripts/push-datasync.sh
If using the pull method, put the following onto each slave web servers crontab:
crontab -e
# Datasync script
*/10 * * * * /opt/scripts/pull-datasync.sh