Copy disk over network using SSH

There can be use cases where you want to get a block level copy of your servers drives either for backup/archival purposes, or maybe you need it for forensic analysis. For both use cases, how can you perform a block level copy of your drives when you don’t have enough storage space on your server to store it?

This is where dd and ssh come into play. You can perform a block level copy of your disks using dd, and then use ssh inline to transfer it over to your remote workstation.

Things to keep in mind, when you perform a dd copy, it will copy the full disk over, this includes both used and unused space. So if you have 2 drives, one is a 1.5T drive, and the other in a 500G drive, you will need a total of 2T of space on your workstation, or where ever the images are being copied to.

In a perfect world, the drives should be unmounted, or if you are copying over the / partition, the server should be in rescue mode. However, dd can be performed live if you like as long as your not concerned about dynamic data like your databases being corrupted on the destination image.

The first step here is to determine which partitions you want to copy over. You find this by checking df on the server:

[root@web01 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                       14G  1.4G   12G  11% /
tmpfs                 939M     0  939M   0% /dev/shm
/dev/sda1             477M   99M  353M  22% /boot
/dev/sdb1             2.0G  3.1M  1.9G   1% /mnt

In this example, I want to copy both my / partition, and my /mnt drive.

The commands are meant to be run on the destination server or workstation where you will be storing these images. Do not try to run these on the origin server!

[root@workstation ~]# ssh [email protected] "dd if=/dev/mapper/VolGroup-lv_root " | dd of=192.168.1.100-VolGroup-lv_root.img
[root@workstation ~]# ssh [email protected] "dd if=/dev/sdb1 " | dd of=192.168.1.100-sdb1.img

Depending on how large the drives are, and what your network latency is like, this could take a very long time to complete.

Once the process is completed, your can verify the image by mounting the image on your destination server by using a loop back image:

[root@workstation ~]# mkdir -p /mnt/server/disk1
[root@workstation ~]# mkdir -p /mnt/server/disk2
[root@workstation ~]# mount -o loop 192.168.1.100-VolGroup-lv_root.img /mnt/server/disk1
[root@workstation ~]# mount -o loop 192.168.1.100-sdb1.img /mnt/server/disk2
[root@workstation ~]# ls /mnt/server/disk1
[root@workstation ~]# ls /mnt/server/disk2

And once you confirmed the disk image looks good, unmount the loop back device by running:

[root@workstation ~]# umount /mnt/server/disk1
[root@workstation ~]# umount /mnt/server/disk2

SSH brute force prevention with fail2ban

Ever take a look at your server’s auth logs and do a quick count of how many failed SSH login attempts you had on your server last week? Its very common to see hundreds, if not thousands of attempts in a very short period of time. Assuming you cannot use a firewall to restrict SSH access to only authorized IP addresses, how do you mitigate these brute force attacks?

There are many tools out there to help with this. One I like is fail2ban. This program scans through log files and takes action against events such as repeated failed login attempts, and blocks the offending IP address for a set period of time.

Procedure

On CentOS systems, fail2ban can be installed from the EPEL repositories. If you do not have EPEL installed, you can get it setup by:

CentOS 5

rpm -ivh http://archives.fedoraproject.org/pub/archive/epel/5/x86_64/epel-release-5-4.noarch.rpm

CentOS 6

rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

CentOS 7

rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-7-11.noarch.rpm

Now install fail2ban:

yum install fail2ban
cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Now customize /etc/fail2ban/jail.local accordingly for your server. Posted below are some more commonly configured options for CentOS 6 servers. The defaults (at the time of this writing) should protect SSH by banning any 3 or greater failed login attempts for 5 minutes via iptables. So the defaults should be okay, but you may want to consider adding your workstations IP address to the ignore ip list below so you don’t lock yourself out by accident!

vi /etc/fail2ban/jail.local

# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not
# ban a host which matches an address in this list. Several addresses can be
# defined using space separator.
ignoreip = 127.0.0.1/8

# "bantime" is the number of seconds that a host is banned.
bantime  = 600

# A host is banned if it has generated "maxretry" during the last "findtime"
# seconds.
findtime  = 600

# "maxretry" is the number of failures before a host get banned.
maxretry = 3

[sshd]

# To use more aggressive sshd filter (inclusive sshd-ddos failregex):
#filter = sshd-aggressive
enabled = true
port    = ssh
logpath = %(sshd_log)s
backend = %(sshd_backend)s

Finally, set fail2ban to start at boot, and start service:

chkconfig fail2ban on
service fail2ban start

As a quick side note, sometimes hosting providers will automatically install fail2ban for you. And depending on the host, they may configure it in such a way that it sends an email each time an IP address gets banned from SSH. This can quickly create a flood of email or email failures, especially if its not configured for a real email address.

If you have having issues like this and your fail2ban configuration was set to email you, you can prevent fail2ban from sending you emails for SSH bans by removing the line from the ssh-iptables block:

sendmail-whois[name=SSH, dest=root, [email protected]]

As a live example assuming it was previously configured, here is what it would look like before you make the change:

vim /etc/fail2ban/jail.conf
...
[ssh-iptables]

enabled  = true
filter   = sshd
action   = iptables[name=SSH, port=ssh, protocol=tcp]
           sendmail-whois[name=SSH, dest=root, [email protected]]
logpath  = /var/log/secure
maxretry = 5

And here is what it would look like after you make the change:

vim /etc/fail2ban/jail.conf
...
[ssh-iptables]

enabled  = true
filter   = sshd
action   = iptables[name=SSH, port=ssh, protocol=tcp]
logpath  = /var/log/secure
maxretry = 5

And be sure to restart fail2ban after the configuration update has been completed:

service fail2ban restart

SSH – Two Factor Authentication

Many people are using Google Authenticator to secure their google apps such as gmail. However what if you wanted to be able to utilize two factor authentication (something you have, something you know) for your SSH logins? What if you want to protect yourself against accidently using weak passwords, which can lead to a successful brute force attack?

On both RedHat and Debian based systems, Google Authenticator’s one time passwords are pretty simple to implement. For the purposes of this guide, I’ll be using CentOS 6 and Ubuntu 12.04.

It should be noted that by using this guide, ALL your users (including root) will be required to use the google authenticator to SSH in unless you have SSH keys already in place. Please check with your administration teams before setting this up to ensure you don’t accidently disable their access, or lock yourself out from SSH!

Procedure

1. Install the module

# RedHat 6 based systems
rpm -ivh http://linux.mirrors.es.net/fedora-epel/6/x86_64/epel-release-6-7.noarch.rpm
yum install google-authenticator

# Debian based systems
aptitude install libpam-google-authenticator

2. Now update the /etc/pam.d/sshd file and add the following at the end of the ‘auth’ section:

auth required pam_google_authenticator.so

3. Then update your /etc/ssh/sshd_config

# Change
ChallengeResponseAuthentication no

# To
ChallengeResponseAuthentication yes

4. Restart sshd

# Redhat:  
service sshd restart

# Ubuntu:  
service ssh restart

5. Now, setup keys for your user

google-authenticator

It will ask you to update your ~/.google_authenticator file, answer yes to this question, and whatever you would like to use for the next three. Once complete, the following will be present to you:

    New Secret Key
    Verification Code
    Emergency Scratch Codes

You will use the new secret key for adding the account to your phone’s google authenticator app. The emergency scratch codes should be copied down and stored somewhere secure. They can be used if you ever lose your iphone, or otherwise need to get into your account without your phone’s google authenticator app.

Now when you log into your server using your user account, it will prompt you for your google auth token, followed by your normal password for the server. Any accounts that don’t have the this setup will not be allowed to log in.

Final thoughts

Remember, two factor authentication is only one part of a defense in depth strategy. No security management system is perfect, but each layer you add will help increase your solutions security footprint.

Using PAM to enforce access based on time

Sometimes there is a need to restrict user access based on time. This could be access to one particular service, or all PAM enabled services. A common example is to only allow access for the user ‘bob’ monday through friday between 9:00AM – 6:00PM. This can be enforced by utilizing the pam_time module.

The pam_time module is an account module type. No arguments are passed directly to the module, but instead all configuration takes place within /etc/security/time.conf.

The time.conf operates based on rules, and each rule uses the following syntax:

services;ttys;users;times

Example Rules
Restrict SSHD access for bob to weekdays between 9:00AM – 7:00PM

sshd;*;bob;Wk0900-1900

Restrict ALL access for bob to weekdays between 9:00AM – 5:00PM

*;*;bob;Wk0900-1700

Restrict ALL access for ALL users except root to weekdays between 9:00AM – 5:00PM

sshd;*;bob;Wk0900-1700

Restrict SSH access for ALL users except bob and jane to weekdays between 9:00AM – 5:00PM

sshd;*;!bob|!jane;Wk0900-1700

To only allow bob to access SSH on Tuesdays between 3:23PM and 4:24PM:

sshd;*;bob;Tu1523-1624

Below is all the available abbreviates for the days of the week:

Mo : Monday Fr : Friday Wd : Sa/Su
Tu : Tuesday Sa : Saturday wk : Mo/Tu/We/Th/Fr
We : Wenesday Su : Sunday
Th : Thursday Al : All Days

Installation And Configuration
In our example, I am going be setting this up on a CentOS 5.x server. For the restricted user, the following variables will be used:

username: bob
allowed access times: 9:00AM - 6:00PM
restricted services: SSHD

First, add the user and time restriction to /etc/security/time.conf:

sshd;*;bob;Wk0900-1800

Now, update the pam module for login and sshd. You are including ‘account required pam_time.so‘. But I’ll post entire file for reference

cat /etc/pam.d/sshd
#%PAM-1.0
auth required pam_sepermit.so
auth include password-auth
account required pam_time.so
account required pam_nologin.so
account include password-auth
password include password-auth
# pam_selinux.so close should be the first session rule
session required pam_selinux.so close
session required pam_loginuid.so
# pam_selinux.so open should only be followed by sessions to be executed in the user context
session required pam_selinux.so open env_params
session optional pam_keyinit.so force revoke
session include password-auth
cat /etc/pam.d/system-auth
#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth required pam_env.so
auth sufficient pam_unix.so nullok try_first_pass
auth requisite pam_succeed_if.so uid >= 500 quiet
auth required pam_deny.so
account required pam_time.so
account required pam_unix.so
account sufficient pam_localuser.so
account sufficient pam_succeed_if.so uid < 500 quiet
account required pam_permit.so
password requisite pam_cracklib.so try_first_pass retry=3 type=
password sufficient pam_unix.so md5 shadow nullok try_first_pass use_authtok
password required pam_deny.so
session optional pam_keyinit.so revoke
session required pam_limits.so
session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session required pam_unix.so

And finally, restart SSH

service sshd restart

SSH Tips and Tricks

SSH is like the Swiss army knife of the unix and linux world. It is a tool that helps solve a variety of issues when it comes to passing traffic over port 22.

This post is going to be a collection of guides showing some of the various solutions that can be accomplished with SSH. This guide assumes you are running Linux on both your office and home workstations.

SSH proxies for web traffic

Suppose you are at a hotel on a public WIFI, and don’t have access to a VPN. How can you still surf the web securely? Assuming you have a server setup at home, or some other secure location, you can simply proxy all your traffic through your home server using an encrypted SSH tunnel.

On your local workstation, run the following:

[user@workstation ~]# ssh -D 7070 [email protected]

Now with the proxy setup, you can route your traffic through a SOCKS 5 proxy in your browsers configuration, or using a plugin such as FoxyProxy. Just be sure that under the SOCKS host, you use:

Host:  127.0.0.1
Port:  7070
Type:  SOCKS v5

SSH proxies for SSH traffic

Using the same scenario, we’re still at the hotel using public WIFI, and don’t have access to a VPN. How can we pass our SSH traffic through a secure proxy server?

On your local workstation, run the following:

[user@workstation ~]# ssh -D 7070 [email protected]

Now configure your .ssh/config to pass all SSH traffic through the SOCKS proxy:

[user@workstation ~]# vim .ssh/config
...
# Filter through socks5 proxy
Host *
ProxyCommand /usr/bin/nc -X 5 -x 127.0.0.1:7070 %h %p
IdentityFile ~/.ssh/id_dsa
ForwardAgent yes
GSSAPIAuthentication no
VerifyHostKeyDNS no
HashKnownHosts no
TCPKeepAlive yes
ServerAliveInterval 300
...

How to X forward applications located on remote servers

Lets suppose you want to access a specific application from your office computer not installed on your home workstation maybe, or want to be able to pick up where you left off on something. For this example, we’ll be accessing Firefox from the office computer, on the home computer.

This would be done on your home computer by:

[user@workstation ~]# ssh -X user@remoteserver firefox

What if you needed to access a internal Windows server from your home computer, but it was only accessible from your office computer? Assuming you can SSH to your office computer, you can run:

[user@workstation ~]# ssh -X user@remoteserver rdesktop -a 24 -u user -f 192.168.1.100

How to create a reverse SSH tunnel

Reverse SSH tunnels are useful when the firewall on the server you need to connect to is blocking inbound SSH connections, but allows for outbound SSH connections. So this is where we simply have the remote server behind the firewall establish the SSH tunnel.

Please understand the security implications of doing this! If you have the ability to use a VPN to access the remote server, use that instead!

On the remote server behind the firewall, create a SSH tunnel to your workstation by:

[root@remoteserver ~]# ssh -R 7022:localhost:22 user@yourworkstation

Now from your workstation, connect to the remote server by:

[user@workstation ~]# ssh -p 7022 localhost

If you want to keep this tunnel up so you don’t have to keep re-establishing the connection when/if it drops, you can script it on the remote server by placing the following script into a cronjob that runs every 5 minutes or something. You will need to have SSH keys in place prior to doing this:

[root@remoteserver ~]# vim /root/ssh-check-tunnel.sh
#!/bin/bash

# Create tunnel if its not established
COMMAND='ssh -N -R 7022:localhost:22 user@yourworkstation'

# Check to see if tunnel is currently up
CHECK_TUNNEL=`ps -eo args | grep "$COMMAND" | grep -v grep`

# If the tunnel is down, create it
if [ -z "$CHECK_TUNNEL" ] ; then
$COMMAND
fi

Then make it executable

[root@remoteserver ~]# chmod 755 /root/ssh-check-tunnel.sh

Finally, drop it into cron:

[root@remoteserver ~]# crontab -e
*/5 * * * * /root/ssh-check-tunnel.sh

How to access servers behind a NAT transparently

Here we will be creating an SSH VPN. However if a regular VPN is available, use that instead. Also understand the security implications of running this before you follow this guide!

To make this a bit easier to understand, here are our devices and IP’s:

workstation:  Ubuntu based workstation
remoterouter:  FreeBSD router running NAT
remoteserver:  Server behind router with a private IP

Assuming you have a workstation running Ubuntu, and you want to be able to access servers that are behind a FreeBSD router running NAT, how can you go about accessing the private IP’s of those remote servers from your workstation simply by running:

[user@workstation ~]# ssh [email protected]

To do this, you need to create a tunnel from your workstation to the FreeBSD router that is providing the private IP’s to the remote servers.

On your workstation, create a SSH key pair without a passphrase (Just take the defaults), and copy that over to the FreeBSD router running NAT:

[root@workstation ~]# ssh-keygen -t dsa
[root@workstation ~]# scp /root/.ssh/id_dsa.pub root@freebsdrouter:/root/.ssh/authorized_keys

Then on your workstation, allow SSHD to create a tunnel:

[root@workstation ~]# vim /etc/ssh/sshd_config
...
PermitTunnel yes
...

Then restart SSH:

[root@workstation ~]# service ssh restart

Now comes the hard part, configure the tunnel interface on your Ubuntu workstation. This guide is assuming 10.1.0.100 and .200 are not in use already on your network. Please be sure to replace ‘remoterouter’ with the IP of your remote server:

[root@workstation ~]# vim /etc/network/interfaces
...
iface tun0 inet static
pre-up ssh -f -w 0:0 root@remoterouter 'ifconfig tun0 10.1.0.200 10.1.0.100'
pre-up sleep 5
address 10.1.0.100
pointopoint 10.1.0.200
netmask 255.255.255.0
up route add -net 10.10.10.0 netmask 255.255.255.0 gw 10.0.0.100 tun0
...

Note, if the remote router is running CentOS 5, you may need to run:

[root@remoterouter ~]# modprobe -av tun
[root@remoterouter ~]# echo modprobe tun >> /etc/rc.modules
[root@remoterouter ~]# chmod -x /etc/rc.modules

On your workstation, bring up the tunnel:

[root@workstation ~]# ifup tun0

Finally, wave a dead chicken over the alter and try to ping one of the servers behind the NAT:

[root@workstation ~]# ping 10.1.0.50

If that works, great! You just created an SSH VPN! You will be able to access any of your servers that are behind this NAT.

If the tunnel gets disconnected, you will need to run the following to re-establish the tunnel:

[root@workstation ~]# ifdown tun0
[root@workstation ~]# ifup tun0