Setting default kernel in grub2

With newer systems like CentOS 7 and Ubuntu 14.04 and 16.04 using Grub2, you can no longer simply update a single file to have your kernel boot off an older or newer kernel. There are a series of steps that must be followed. The examples below will show how to boot off an older kernel for their respective operating systems.

Please note, the instructions in this article will lock your kernel on whichever one you selected. Even if your system receives automatic kernel updates, those new kernels will have to be manually enabled within grub if you want to use them.

CentOS 7

First, check to see which kernel is currently running:

[root@web01 ~]# uname -r
3.10.0-514.16.1.el7.x86_64

That shows us we’re running 3.10.0-514.16.1, however I need to be running 3.10.0-327.36.3. So to use this specific named kernel, first changed the GRUB_DEFAULT to ‘saved’ in /etc/default/grub by:

[root@web01~]# cp /etc/default/grub /etc/default/grub.bak
[root@web01~]# vim /etc/default/grub
...
GRUB_DEFAULT=saved
...

Now create a backup of the grub config for recovery purposes if needed, then rebuild grub:

[root@web01 ~]# cp /boot/grub2/grub.cfg /boot/grub2/grub.cfg.bak
[root@web01 ~]# grub2-mkconfig -o /boot/grub2/grub.cfg

Determine what the full kernel name is you want to use. Get the listing by running:

[root@web01~]# grep "^menuentry" /boot/grub2/grub.cfg | cut -d "'" -f2
CentOS Linux (3.10.0-514.16.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-514.2.2.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-327.36.3.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-327.22.2.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-327.3.1.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-c11d017c89ca4e8685ae3d9791d472ca) 7 (Core)

I want to use the 3.10.0-327.36.3 kernel. Setting that to be the default kernel is simple. Set the desired kernel by running:

[root@web01 ~]# grub2-set-default "CentOS Linux (3.10.0-327.36.3.el7.x86_64) 7 (Core)"

Now verify that the change got applied in the configs by running:

[root@web01 ~]# grub2-editenv list
saved_entry=CentOS Linux (3.10.0-327.36.3.el7.x86_64) 7 (Core)

Reboot the system so it boots off the older kernel:

root@web01:~# reboot

Finally, once the system comes back online, verify the desired kernel is running by:

[root@web01 ~]# uname -r
3.10.0-327.36.3.el7.x86_64

If the system rebooted, and dropped you into a grub shell with an error, you can boot up off of the backup grub.cfg file that was created by:

grub2> configfile (hd0,1)/boot/grub2/grub.cfg.bak

Ubuntu 14.04 and Ubuntu 16.04

First, check to see which kernel is currently running:

root@web01:~# uname -r
4.4.0-48-generic

That shows us we’re running 4.4.0-48, however I need to be running 4.4.0-47. So to use this specific named kernel, first changed the GRUB_DEFAULT to ‘saved’ in /etc/default/grub by:

root@web01:~# cp /etc/default/grub /etc/default/grub.bak
root@web01:~# vim /etc/default/grub
...
GRUB_DEFAULT=saved
...

Now create a backup of the grub config for recovery purposes if needed, then rebuild grub:

root@web01:~# cp /boot/grub/grub.cfg /boot/grub/grub.cfg.bak
root@web01:~# update-grub

Determine what the full kernel name is you want to use. Get the listing by running:

root@web01:~# egrep "^[[:space:]]?(submenu|menuentry)" /boot/grub/grub.cfg | cut -d "'" -f2
Ubuntu
Advanced options for Ubuntu
Ubuntu, with Linux 4.4.0-78-generic
Ubuntu, with Linux 4.4.0-78-generic (recovery mode)
Ubuntu, with Linux 4.4.0-47-generic
Ubuntu, with Linux 4.4.0-47-generic (recovery mode)

I want to use the 4.4.0-47-generic kernel. Setting that to be the default kernel is simple. However, you MUST prepend ‘Advanced options for Ubuntu’ to the kernel name as shown below since Ubuntu makes use of sub menus in the kernel listing. So set the desired kernel by running:

root@web01:~# grub-set-default 'Advanced options for Ubuntu>Ubuntu, with Linux 4.4.0-47-generic'

Now verify that the change got applied in the configs by running:

root@web01:~# grub-editenv list
saved_entry=Advanced options for Ubuntu>Ubuntu, with Linux 4.4.0-47-generic

Reboot the system so it boots off the older kernel:

root@web01:~# reboot

Finally, once the system comes back online, verify the desired kernel is running by:

root@web01:~# uname -r
4.4.0-47-generic

If the system rebooted, and dropped you into a grub shell with an error, you can boot up off of the backup grub.cfg file that was created by:

grub2> configfile (hd0,1)/boot/grub2/grub.cfg.bak

Setting up the old-releases repo for ubuntu

This is a guide on enabling the old-releases repos for the Ubuntu project. When Ubuntu marks a release EOL, the mainline repos are moved to an alternate location where they are preserved for historical purposes. It should be noted that when a system reaches EOL, the repos for it are no longer maintained.

It goes without saying that enabling the old-releases repos for EOL systems should only be used as a last resort. Continuing to run a system that has reached the end of life status is dangerous as it no longer receives security patches and bug fixes. There are also no promises that things will continue to work.

Mitigating security and reliability issues can be resolved all together by updating the system to a supported version of the operating system. If there is a compelling reason to enable the old-releases repos so packages can be installed, then proceed below.

Ubuntu 10.04 EOL Repos

Ubuntu 10.04 LTS went EOL on 4/2015. The procedure for setting up the system to use the old-releases repos are below.

First, create a backup of the /etc/apt/sources.list by:

[root@web01 ~]# cp /etc/apt/sources.list /etc/apt/sources.list.bak

Now update /etc/apt/sources.list to point to the old-releases repos accordingly. Keep in mind that there may be repos specified in here for Nginx, Varnish, Docker, etc. So be sure to only update the items needed by Ubuntu. The end result should look something like this:

[root@web01 ~]# vim /etc/apt/sources.list
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.

deb http://old-releases.ubuntu.com/ubuntu/ lucid main restricted
deb-src http://old-releases.ubuntu.com/ubuntu/ lucid main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://old-releases.ubuntu.com/ubuntu/ lucid-updates main restricted
deb-src http://old-releases.ubuntu.com/ubuntu/ lucid-updates main restricted

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://old-releases.ubuntu.com/ubuntu/ lucid universe
deb-src http://old-releases.ubuntu.com/ubuntu/ lucid universe
deb http://old-releases.ubuntu.com/ubuntu/ lucid-updates universe
deb-src http://old-releases.ubuntu.com/ubuntu/ lucid-updates universe

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu 
## team, and may not be under a free licence. Please satisfy yourself as to 
## your rights to use the software. Also, please note that software in 
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://old-releases.ubuntu.com/ubuntu/ lucid multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ lucid multiverse
deb http://old-releases.ubuntu.com/ubuntu/ lucid-updates multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ lucid-updates multiverse

## Uncomment the following two lines to add software from the 'backports'
## repository.
## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
# deb http://old-releases.ubuntu.com/ubuntu/ lucid-backports main restricted universe multiverse
# deb-src http://old-releases.ubuntu.com/ubuntu/ lucid-backports main restricted universe multiverse

## Uncomment the following two lines to add software from Canonical's
## 'partner' repository.
## This software is not part of Ubuntu, but is offered by Canonical and the
## respective vendors as a service to Ubuntu users.
# deb http://old-releases.ubuntu.com/ubuntu lucid partner
# deb-src http://old-releases.ubuntu.com/ubuntu lucid partner

deb http://old-releases.ubuntu.com/ubuntu lucid-security main restricted
deb-src http://old-releases.ubuntu.com/ubuntu lucid-security main restricted
deb http://old-releases.ubuntu.com/ubuntu lucid-security universe
deb-src http://old-releases.ubuntu.com/ubuntu lucid-security universe
deb http://old-releases.ubuntu.com/ubuntu lucid-security multiverse
deb-src http://old-releases.ubuntu.com/ubuntu lucid-security multiverse

Now refresh the package index from their sources by running:

[root@web01 ~]# apt-get update

Address any 404’s accordingly as that means the URL may be incorrect or may not longer exist.

Ubuntu 12.04 EOL Repos

Ubuntu 12.04 LTS is going EOL on 4/2017. The procedure for setting up the system to use the old-releases repos are below.

First, create a backup of the /etc/apt/sources.list by:

[root@web01 ~]# cp /etc/apt/sources.list /etc/apt/sources.list.bak

Now update /etc/apt/sources.list to point to the old-releases repos accordingly. Keep in mind that there may be repos specified in here for Nginx, Varnish, Docker, etc. So be sure to only update the items needed by Ubuntu. The end result should look something like this:

[root@web01 ~]# vim /etc/apt/sources.list
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://old-releases.ubuntu.com/ubuntu/ precise main restricted
deb-src http://old-releases.ubuntu.com/ubuntu/ precise main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://old-releases.ubuntu.com/ubuntu/ precise-updates main restricted
deb-src http://old-releases.ubuntu.com/ubuntu/ precise-updates main restricted

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://old-releases.ubuntu.com/ubuntu/ precise universe
deb-src http://old-releases.ubuntu.com/ubuntu/ precise universe
deb http://old-releases.ubuntu.com/ubuntu/ precise-updates universe
deb-src http://old-releases.ubuntu.com/ubuntu/ precise-updates universe

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu 
## team, and may not be under a free licence. Please satisfy yourself as to 
## your rights to use the software. Also, please note that software in 
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://old-releases.ubuntu.com/ubuntu/ precise multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ precise multiverse
deb http://old-releases.ubuntu.com/ubuntu/ precise-updates multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ precise-updates multiverse

## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
deb http://old-releases.ubuntu.com/ubuntu/ precise-backports main restricted universe multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ precise-backports main restricted universe multiverse

deb http://old-releases.ubuntu.com/ubuntu precise-security main restricted
deb-src http://old-releases.ubuntu.com/ubuntu precise-security main restricted
deb http://old-releases.ubuntu.com/ubuntu precise-security universe
deb-src http://old-releases.ubuntu.com/ubuntu precise-security universe
deb http://old-releases.ubuntu.com/ubuntu precise-security multiverse
deb-src http://old-releases.ubuntu.com/ubuntu precise-security multiverse

## Uncomment the following two lines to add software from Canonical's
## 'partner' repository.
## This software is not part of Ubuntu, but is offered by Canonical and the
## respective vendors as a service to Ubuntu users.
# deb http://old-releases.ubuntu.com/ubuntu precise partner
# deb-src http://old-releases.ubuntu.com/ubuntu precise partner

## Uncomment the following two lines to add software from Ubuntu's
## 'extras' repository.
## This software is not part of Ubuntu, but is offered by third-party
## developers who want to ship their latest software.
# deb http://old-releases.ubuntu.com/ubuntu precise main
# deb-src http://old-releases.ubuntu.com/ubuntu precise main

Now refresh the package index from their sources by running:

[root@web01 ~]# apt-get update

Address any 404’s accordingly as that means the URL may be incorrect or may not longer exist.

Using hpasmcli for HP servers

HP comes with their server utility scripts called hpssacli and hpacucli. These tools allow you to view and modify your hardware configuration on the server. The hpacucli is the older implementation of the toolkit, but the syntax is pretty similar.

HP tools information

To show the firmware version, run:

[root@web01 ~]# hpasmcli -s "show server"

If you want to see extended information, run:

[root@web01 ~]# hpssacli controller all show config detail

General information

To view information regarding the server model, cpu, type, memory, etc, run:

[root@web01 ~]# hpasmcli -s "show server"

Hardware Health

If you want to view the health of the system and chassis components, run:

[root@web01 ~]# hpasmcli -s "show server"

The chassis can also only return specific components, such as:

[root@web01 ~]# hpasmcli -s "show powersupply"
[root@web01 ~]# hpasmcli -s "show dimm"
[root@web01 ~]# hpasmcli -s "show fans"
[root@web01 ~]# hpasmcli -s "show temp"

Storage health

To view the physical and virtual disks on the server:

[root@web01 ~]# hpssacli ctrl all show config
[root@web01 ~]# hpssacli controller slot=3 physicaldrive all show
[root@web01 ~]# hpssacli controller slot=3 physicaldrive 2I:1:5 show detail
[root@web01 ~]# hpssacli controller slot=3 logicaldrive all show

On older HP servers, you can view the physical and virtual disks on the server by:

[root@web01 ~]# hpacucli controller slot=1 physicaldrive all show
[root@web01 ~]# hpacucli controller slot=1 physicaldrive 2I:1:5 show detail
[root@web01 ~]# hpacucli controller slot=1 logicaldrive all show

To see the storage battery status:

[root@web01 ~]# hpssacli controller all show detail | egrep -i 'battery\/|controller\ status|cache\ status'
   Controller Status: OK
   Cache Status: OK
   Battery/Capacitor Count: 1
   Battery/Capacitor Status: OK

Hardware logs

To display the hardware logs:

[root@web01 ~]# hpasmcli -s "show iml"

If you need to clear the hardware logs:

[root@web01 ~]# hpasmcli -s "clear iml"

CPU actions

To see if hyperthreading is enabled on the CPUs:

[root@web01 ~]# hpasmcli -s "show ht"

If you wanted to change the hyperthreading settings:

# Enable
[root@web01 ~]# hpasmcli -s "enable ht"

# Disable
[root@web01 ~]# hpasmcli -s "disable ht"

Using omreport and omconfig for Dell servers

Dell comes with their server utility scripts called omreport and omconfig. These tools allow you to view and modify your hardware configuration on the server.

Dell tools information

To see what version of the tools your running:

[root@web01 ~]# omreport about details=true

To see if there are updates available for the firmware:

[root@web01 ~]# omreport system version

To see what commands are available using omreport:

[root@web01 ~]# omreport system -?

General information

To view information regarding the server model, cpu type, memory, service tags, etc, run:

[root@web01 ~]# omreport system summary

Hardware Health

If you want to view the health of the system and chassis components, run:

[root@web01 ~]# omreport system

To only get the health information for the chassis:

[root@web01 ~]# omreport chassis

The chassis can also only return specific components, such as:

[root@web01 ~]# omreport chassis fans
[root@web01 ~]# omreport chassis memory
[root@web01 ~]# omreport chassis nics
[root@web01 ~]# omreport chassis processors
[root@web01 ~]# omreport chassis temps
[root@web01 ~]# omreport chassis batteries
[root@web01 ~]# omreport chassis pwrsupplies

Storage health

As a quick note, if the commands below report there are no controllers listed, check to be sure that the software is actually running by:

[root@web01 ~]# /opt/dell/srvadmin/sbin/srvadmin-services.sh status
dell_rbu (module) is stopped
ipmi driver is running
dsm_sa_datamgrd is stopped
dsm_sa_eventmgrd is stopped
dsm_sa_snmpd is stopped
dsm_om_shrsvcd is stopped
dsm_om_connsvcd is stopped
[root@web01 ~]# /opt/dell/srvadmin/sbin/srvadmin-services.sh restart

To view the physical and virtual disks on the server:

[root@web01 ~]# omreport storage pdisk controller=0
[root@web01 ~]# omreport storage vdisk controller=0
[root@web01 ~]# omreport storage pdisk controller=0 vdisk=0

If you just wanted a quick listing of the relevant disk information to see the state of the drives, run:

[root@web01 ~]# omreport storage pdisk controller=0 | grep -iE "^id|^status|name|state|Failure Predicted"
ID                              : 0:0:0
Status                          : Ok
Name                            : Physical Disk 0:0:0
State                           : Online
Failure Predicted               : No
ID                              : 0:0:1
Status                          : Ok
Name                            : Physical Disk 0:0:1
State                           : Online
Failure Predicted               : No

To see if there are any empty drive bays on the server:

[root@web01 ~]# omreport storage controller controller=0 info=pdslotreport | grep 'Empty Slots'

To see the storage battery status:

[root@web01 ~]# omreport storage battery controller=0

Hardware Logs

To display the hardware logs, run:

[root@web01 ~]# omreport system esmlog

If you need to view the alert logs:

[root@web01 ~]# omreport system alertlog

And if you needed to view the messages from the POST:

[root@web01 ~]# omreport system postlog

If you find you need to clear the logs, that can be performed by:

[root@web01 ~]# omconfig system esmlog action=clear
[root@web01 ~]# omconfig system alertlog action=clear
[root@web01 ~]# omconfig system postlog action=clear

CPU actions

To see if hyperthreading is enabled on the CPUs:

[root@web01 ~]# omreport chassis biossetup | grep -A 2 'HyperThreading'

If you wanted to enable hyperthreading:

# Dell R710
[root@web01 ~]# omconfig chassis biossetup attribute=cpuht setting=enabled

# Dell R720
[root@web01 ~]# omconfig chassis biossetup attribute=ProcCores setting=All

If you needed to enable or disable NUMA:

# Disable NUMA:
[root@web01 ~]# omconfig chassis biossetup attribute=numa setting=disabled

# Enable NUMA:
[root@web01 ~]# omconfig chassis biossetup attribute=numa setting=enabled

Changing your servers timezone

The systems timezone is usually set during installation. If the timezone needed to be changed, it can be done without rebooting the system.

Just be sure to watch for applications like MySQL and PHP that require additional steps for changing the timezone, which are noted near the bottom of this article.

CentOS 5 and CentOS 6

Modify the zone in /etc/sysconfig/clock. You can find the valid timezones in /usr/share/zoneinfo. The commonly used timezones include America/New_York, America/Chicago, America/Los_Angeles, and UTC.

[root@web01 ~]# vim /etc/sysconfig/clock
...
ZONE="America/New_York"
...

Then update /etc/localtime:

[root@web01 ~]# tzdata-update

Now sync the hardware clock against the system time:

[root@web01 ~]# hwclock --systohc

Go ahead and restart syslogd/rsyslogd and crond:

[root@web01 ~]# service crond restart
[root@web01 ~]# service rsyslog restart
[root@web01 ~]# service syslog restart

CentOS 7

Changing the timezone on CentOS 7 can be done with a few commands. You can find the valid timezones in /usr/share/zoneinfo. The commonly used timezones include America/New_York, America/Chicago, America/Los_Angeles, and UTC.

[root@web01 ~]# timedatectl set-timezone America/New_York
[root@web01 ~]# systemctl restart crond
[root@web01 ~]# systemctl restart rsyslog

Ubuntu 12.04 and Ubuntu 14.04

Modify the zone in /etc/timezone. You can find the valid timezones in /usr/share/zoneinfo. The commonly used timezones include America/New_York, America/Chicago, America/Los_Angeles, and UTC.

[root@web01 ~]# vim /etc/timezone
...
America/New_York
...

Now update active timezone:

[root@web01 ~]# dpkg-reconfigure --frontend noninteractive tzdata
Current default time zone: 'America/New_York'
Local time is now:      Tue Jan 17 01:18:04 EST 2017.
Universal Time is now:  Tue Jan 17 06:18:04 UTC 2017.

Restart rsyslog and cron:

[root@web01 ~]# service cron restart
[root@web01 ~]# service rsyslog restart

Ubuntu 16.04

Changing the timezone on Ubuntu 16.04 can be done with a few commands. You can find the valid timezones in /usr/share/zoneinfo. The commonly used timezones include America/New_York, America/Chicago, America/Los_Angeles, and UTC.

[root@web01 ~]# timedatectl set-timezone America/New_York
[root@web01 ~]# systemctl restart crond
[root@web01 ~]# systemctl restart rsyslog

MySQL, Percona and MariaDB

In order for MySQL, Percona, and MariaDB to register the new timezone settings, they need to be restarted. There really isn’t a way around this. As a temporary solution, and one that will not pick up future DST timezone changes, you can manually set the databases current time by:

# List current system date and time via MySQL:
[root@web01 ~]# mysql
mysql> SELECT @@global.time_zone;

# List the current date and time according to MySQL:
mysql> SELECT NOW();

# Update the timezone using the UTC offset:
mysql> SET @@global.time_zone = '+05:00';

Even with this temporary fix in place, unless you are using the UTC timezone, MySQL should be restarted very soon.

PHP

PHP should also have its timezone updated when you change it on the system. Determine where your php.ini resides, then update it accordingly. I am assuming CentOS 6 for this example:

[root@web01 ~]# vim /etc/php.ini
...
date.timezone = "America/New_York"
...

Then restart Apache or PHP-FPM for your system so the changes are applied. Afterwards, test the timezone change to PHP by:

[root@web01 ~]# php -i |grep date.timezone
date/time support => enabled
date.timezone => America/New_York => America/New_York

A list of supported timezone configurations for PHP can be found at:
http://www.php.net/manual/en/timezones.php

Changing the servers hostname

The hostname of a system is usually set during installation. However as time goes on, it may be determined that the hostname should be changed to something more suitable.

Outlined below is the procedure for changing the systems hostname without having to reboot.

CentOS 5 and 6

First, check to see what the existing hostname is on the server:

[root@web01 ~]# hostname

Then change the current hostname. In this example, we’re going to change the hostname to web04.mydomain.com:

[root@web01 ~]# hostname web04.mydomain.com

Now update /etc/hosts with the new hostname:

[root@web01 ~]# vim /etc/hosts
127.0.0.1      localhost.localdomain.com localhost
192.168.1.5    web04.mydomain.com web04

If you have ‘domain’ specified in /etc/resolv.conf, update that accordingly. There is not a need to add ‘domain’ if it is not already defined:

[root@web01 ~]# vim /etc/resolv.conf
domain web04.mydomain.com
nameserver 8.8.8.8
nameserver 8.8.4.4

Next update your network configuration:

[root@web01 ~]# vim /etc/sysconfig/network
...
HOSTNAME=web04.mydomain.com
...

Restart syslog so the new changes go into effect:

# CentOS 5
[root@web01 ~]# service syslog restart

# CentOS 6
[root@web01 ~]# service rsyslog restart

Finally, log out and back into the server via SSH and you should see the new hostname in effect. Also check /var/log/secure and /var/log/messages to ensure the new hostname is being used.

CentOS 7

In this example, we’re going to change the hostname to web04.mydomain.com. First run:

[root@web01 ~]# hostname web04.mydomain.com

Now update /etc/hosts with the new hostname:

[root@web01 ~]# vim /etc/hosts
127.0.0.1      localhost.localdomain.com localhost
192.168.1.5    web04.mydomain.com web04

Now update the hostname using the systemd command:

[root@web01 ~]# hostnamectl set-hostname web04.mydomain.com

If you have ‘domain’ specified in /etc/resolv.conf, update that accordingly. There is not a need to add ‘domain’ if it is not already defined:

[root@web01 ~]# vim /etc/resolv.conf
domain web04.mydomain.com
nameserver 8.8.8.8
nameserver 8.8.4.4

Now restart syslog so the new changes go into effect:

[root@web01 ~]# systemctl restart rsyslog

Finally, log out and back into the server via SSH and you should see the new hostname in effect. Also check /var/log/auth.log and /var/log/syslog to ensure the new hostname is being used.

Ubuntu 12.04 and Ubuntu 14.04

First, check to see what the existing hostname is on the server:

[root@web01 ~]# hostname

In this example, we’re going to change the hostname to web04.mydomain.com. So run:

[root@web01 ~]# hostname web04.mydomain.com

Now update /etc/hosts with the new hostname:

[root@web01 ~]# vim /etc/hosts
127.0.0.1      localhost.localdomain.com localhost
192.168.1.5    web04.mydomain.com web04

If you have ‘domain’ specified in /etc/resolv.conf, update that accordingly. There is not a need to add ‘domain’ if it is not already defined:

[root@web01 ~]# vim /etc/resolv.conf
domain web04.mydomain.com
nameserver 8.8.8.8
nameserver 8.8.4.4

Then update /etc/hostname accordingly:

[root@web01 ~]# vim /etc/hostname
web04.mydomain.com

Now restart syslog so the new changes go into effect:

[root@web01 ~]# service rsyslog restart

Finally, log out and back into the server via SSH and you should see the new hostname in effect. Also check /var/log/auth.log and /var/log/syslog to ensure the new hostname is being used.

Ubuntu 16.04 and Ubuntu 18.04

First, check to see what the existing hostname is on the server:

[root@web01 ~]# hostname

In this example, we’re going to change the hostname to web04.mydomain.com. So run:

[root@web01 ~]# hostname web04.mydomain.com

Now update /etc/hosts with the new hostname:

[root@web01 ~]# vim /etc/hosts
127.0.0.1      localhost.localdomain.com localhost
192.168.1.5    web04.mydomain.com web04

Now update the hostname using the systemd command:

[root@web01 ~]# hostnamectl set-hostname web04.mydomain.com

If you have ‘domain’ specified in /etc/resolv.conf, update that accordingly. There is not a need to add ‘domain’ if it is not already defined:

[root@web01 ~]# vim /etc/resolv.conf
domain web04.mydomain.com
nameserver 8.8.8.8
nameserver 8.8.4.4

Now restart syslog so the new changes go into effect:

[root@web01 ~]# systemctl restart rsyslog

On Ubuntu 18.04 only, check to see if /etc/cloud/cloud.cfg exists. If it does, confirm the preserve hostname is set to true as shown below:

[root@web01 ~]# vim /etc/cloud/cloud.cfg
...
preserve_hostname: true
...

Finally, log out and back into the server via SSH and you should see the new hostname in effect. Also check /var/log/auth.log and /var/log/syslog to ensure the new hostname is being used.

Using access control lists (FACLs)

An access control list (ACL) provides a more granular approach to permissions, allowing the system administrator the ability to move past the limitations of Linux permissions when the situation warrants it.

For instance, perhaps you have a complex use case that requires granting permissions to multiple individual users, or maybe to more then a single group. Some people would get around this by simply using 777 permissions, but that in and of itself is a very poor security practice.

ACLs can help solve this as they allow you to create a sophisticated permissions scheme that grant access to the users and groups that require it, without the need to open the permissions broadly for everyone.

With all this being said, ACLs should only be used when standard permissions cannot accomplish what your looking to do. ACLs are often used as the ‘quick fix’, but while they do work, they can overly complicate an environment that could cause major headaches while troubleshooting. There is also more room for error, as you are adding an additional layer of complication when employing ACLs.

Enabling ACLs on the filesystem

Setting up a system for ACLs are pretty simple. If you are running a system that uses systemd, then ACLs should already been enabled as its a dependency for systemd. For older systems, the process is shown below for installing and enabling ACLs.

First, check to see if ACLs are already enabled on the filesystem:

[root@web01 ~]# tune2fs -l /dev/sda1 |grep acl
Default mount options:    user_xattr acl

If you do not see ‘acl’ in the output, then you can install ACLs by:

# CentOS / RHEL:
[root@web01 ~]# yum install acl

# Ubuntu / Debian:
[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install acl

Now for all distros, enable ACLs in your /etc/fstab putting ‘acl’ in the mount opens as shown below:

[root@web01 ~]# cat /etc/fstab 
/dev/mapper/VolGroup-lv_root / ext4 defaults,acl 1 1

Then remount the filesystem so the new mount option takes effect by:

[root@web01 ~]# mount -o remount /

Then verify that ACL’s are now enabled on the filesystem:

[root@web01 ~]# tune2fs -l /dev/sda1 |grep acl
Default mount options:    user_xattr acl

Using ACLs

You can determine if a file or directory has an ACL in place as the permissions will be followed by a ‘+’ as shown below:

[root@web01 ~]#  ls -al
...
-rw-rwxr--+  1 root root   93 Jan 12 11:22 test
...

You can see what ACLs have been assigned to the file by:

[root@web01 ~]# getfacl test 
# file: test
# owner: root
# group: root
user::rw-
user:jdoe:rwx
group::r--
mask::rwx
other::r--

So in the example above, we see a user called jdoe also rwx permissions to the file called test.

To add or modify a user ACL:

[root@web01 ~]# setfacl -m u:jdoe:permissions /var/www/vhosts/domain.com

To add or modify a group ACL:

[root@web01 ~]# setfacl -m g:devs:permissions /var/www/vhosts/domain.com

To modify the ‘other’ ACL:

[root@web01 ~]# setfacl -m o:permissions /var/www/vhosts/domain.com

To add or modify an ACL recursively, you use the -R option. It is important to note that it is considered good practice to use the -X (capital X) permission when using recursion so files can retain the execute permission if they have them, but also allow you to traverse the directory as that requires the execute permission. The benefit comes into play with the -X (capital X) as it will prevent an admin from accidentally adding the execute permission on a regular file. An example of this is below:

[root@web01 ~]# setfacl -R -m u:jdoe:rwX /var/www/vhosts/domain.com

To set the default ACL so all new directories will inherit the ACL’s set on the parent directory:

[root@web01 ~]# setfacl -R -m default:u:jdoe:rwX /var/www/vhosts/domain.com

To remove a user from an ACL:

[root@web01 ~]# setfacl -x u:jdoe /var/www/vhosts/domain.com

To remove a group from an ACL:

[root@web01 ~]# setfacl -x g:devs /var/www/vhosts/domain.com

Remove ALL ACL’s on a file/directory:

[root@web01 ~]# setfacl -b /var/www/vhosts/domain.com

Remove ALL ACL’s on a file/directory recursively:

[root@web01 ~]# setfacl -R -b /var/www/vhosts/domain.com

Quick programming language test scripts

This article simply contains how to test a few languages with a hello world script and a way to test mail functionality.

PHP

PHP script that will output hello world:

[root@web01 ~]# vim hello_world.php
#!/usr/bin/php
<?php
print "Hello World!\n";
?>

[root@web01 ~]# php hello_world.php 
Hello World!

PHP script to test mail functionality:

[root@web01 ~]# vim email_test.php
#!/usr/bin/php
<?php
$hostname = shell_exec('hostname');
$to = "[email protected]";
$subject = "Test mail";
$message = "Hello! This is a simple email message.";
$from = "root@" . $hostname;
$headers = "From:" . $from;
$parameters = "-f " . $from;
mail($to,$subject,$message,$headers,$parameters);
echo "Mail Sent.\n";
?>

[root@web01 ~]# php email_test.php
Mail Sent.

PHP script to test MySQL access:

[root@web01 ~]# db_test.php
<?php
# Fill our vars and run on cli
$dbname = 'DATABASE_NAME';
$dbuser = 'DATABASE_USER';
$dbpass = 'DATABASE_PASS';
$dbhost = 'DATABASE_HOST';
$link = mysqli_connect($dbhost, $dbuser, $dbpass) or die("Unable to Connect to '$dbhost'");
mysqli_select_db($link, $dbname) or die("Could not open the db '$dbname'");
$test_query = "SHOW TABLES FROM $dbname";
$result = mysqli_query($link, $test_query);
$tblCnt = 0;
while($tbl = mysqli_fetch_array($result)) {
$tblCnt++;
}
if (!$tblCnt) {
echo "There are no tables<br />\n";
} else {
echo "There are $tblCnt tables<br />\n";
}

[root@web01 ~]# php db_test.php
There are no tables

Perl

Perl script that will output hello world:

[root@web01 ~]# vim hello_world.pl
#!/usr/bin/perl
print "Hello World!\n";

[root@web01 ~]# perl hello_world.pl
Hello World!

Perl script to test mail functionality:

[root@web01 ~]# vim email_test.pl
#!/usr/bin/perl
use Sys::Hostname;
$hostname = hostname;
$to = '[email protected]';
$subject = 'Test mail';
$message = 'Hello! This is a simple email message.';
$from = 'root@' . $hostname;
open (MAIL,'|/usr/sbin/sendmail -t');
print MAIL "To: $to\n";
print MAIL "From: $from\n";
print MAIL "Subject: $subject\n\n";
print MAIL $message;
close (MAIL);
print "Mail Sent.\n";

[root@web01 ~]# perl email_test.pl
Mail Sent.

Python
Python script that will output hello world:

[root@web01 ~]# vim hello_world.py
#!/usr/bin/python
print "Hello World!";

[root@web01 ~]# python hello_world.py
Hello World!

Python script to test mail functionality:

[root@web01 ~]# vim email_test.py
#!/usr/bin/python
import smtplib
import socket
hostname = socket.getfqdn()
mailfrom = 'root@' + hostname
to = '[email protected]'
subject = 'Test mail'
message = 'Hello! This is a simple email message.'
smtpserver = smtplib.SMTP("127.0.0.1",25)
header = 'To:' + to + '\n' + 'From: ' + mailfrom + '\n' + 'Subject:' + subject + '\n\n'
smtpserver.sendmail(mailfrom, to, header + message)
print 'Mail Sent.'
smtpserver.close()

[root@web01 ~]# python email_test.py
Mail Sent.

Ruby

Ruby script that will output hello world:

[root@web01 ~]# vim hello_world.rb
#!/usr/bin/ruby
puts 'Hello World!'

[root@web01 ~]# ruby hello_world.rb
Hello World!

Ruby script to test mail functionality:

[root@web01 ~]# vim email_test.rb
#!/usr/bin/ruby
require 'net/smtp'
require 'socket'
hostname = Socket.gethostname
from = 'root@' + hostname
to = '[email protected]'
subject = 'Test mail'
message = 'Hello! This is a simple email message.'
msg = <<EOF
From: #{from}
To: #{to}
Subject: #{subject}
#{message}
EOF
Net::SMTP.start('localhost').send_message msg, from, to
puts 'Mail Sent.'

[root@web01 ~]# ruby email_test.rb
Mail Sent.

Copy disk over network using SSH

There can be use cases where you want to get a block level copy of your servers drives either for backup/archival purposes, or maybe you need it for forensic analysis. For both use cases, how can you perform a block level copy of your drives when you don’t have enough storage space on your server to store it?

This is where dd and ssh come into play. You can perform a block level copy of your disks using dd, and then use ssh inline to transfer it over to your remote workstation.

Things to keep in mind, when you perform a dd copy, it will copy the full disk over, this includes both used and unused space. So if you have 2 drives, one is a 1.5T drive, and the other in a 500G drive, you will need a total of 2T of space on your workstation, or where ever the images are being copied to.

In a perfect world, the drives should be unmounted, or if you are copying over the / partition, the server should be in rescue mode. However, dd can be performed live if you like as long as your not concerned about dynamic data like your databases being corrupted on the destination image.

The first step here is to determine which partitions you want to copy over. You find this by checking df on the server:

[root@web01 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                       14G  1.4G   12G  11% /
tmpfs                 939M     0  939M   0% /dev/shm
/dev/sda1             477M   99M  353M  22% /boot
/dev/sdb1             2.0G  3.1M  1.9G   1% /mnt

In this example, I want to copy both my / partition, and my /mnt drive.

The commands are meant to be run on the destination server or workstation where you will be storing these images. Do not try to run these on the origin server!

[root@workstation ~]# ssh [email protected] "dd if=/dev/mapper/VolGroup-lv_root " | dd of=192.168.1.100-VolGroup-lv_root.img
[root@workstation ~]# ssh [email protected] "dd if=/dev/sdb1 " | dd of=192.168.1.100-sdb1.img

Depending on how large the drives are, and what your network latency is like, this could take a very long time to complete.

Once the process is completed, your can verify the image by mounting the image on your destination server by using a loop back image:

[root@workstation ~]# mkdir -p /mnt/server/disk1
[root@workstation ~]# mkdir -p /mnt/server/disk2
[root@workstation ~]# mount -o loop 192.168.1.100-VolGroup-lv_root.img /mnt/server/disk1
[root@workstation ~]# mount -o loop 192.168.1.100-sdb1.img /mnt/server/disk2
[root@workstation ~]# ls /mnt/server/disk1
[root@workstation ~]# ls /mnt/server/disk2

And once you confirmed the disk image looks good, unmount the loop back device by running:

[root@workstation ~]# umount /mnt/server/disk1
[root@workstation ~]# umount /mnt/server/disk2

Determine how sites react under increased requests with curl

There are times when you test out your site using curl, and it loads fine. But yet when multiple users or a search bot is going through your site, you notice the page request times skyrocket. There could be many causes, though usually it revolves around a lack of caching since repeated requests for the same page need to first check in with the database, then get passed over to PHP to finish processing the request.

Okay great, cache is king. We know this. But how can we actually confirm this is a problem? Well, this is where curl comes to the rescue.

First, establish a baseline by sending a single request to each website:

[user@workstation01 ~]# for i in www.example1.com www.example2.com www.example3.com; do echo -n "$i "; (time curl -IL $i -XGET) 2>&1 | grep -E "real|HTTP"; echo; done

www.example1.com HTTP/1.1 200 OK
real	0m0.642s

www.example2.com HTTP/1.1 200 OK
real	0m2.234s

www.example3.com HTTP/1.1 200 OK
real	0m0.421s

So based off those results, we established that 2 of the sites take about half a second to load, and the other one (www.example2.com) takes about 2 seconds to load when hit with a single request.

As www.example2.com takes 2 seconds to load with a single request, lets see what happens during increased traffic. This could be from valid users, search bots, or something else. How much will the load times increase for the site? Here is an example sending over 25 requests:

[user@workstation01 ~]# for i in {1..25}; do (time curl -sIL http://www.example2.com -X GET &) 2>&1 | grep -E "real|HTTP" & done

HTTP/1.1 200 OK
real	0m11.297s
HTTP/1.1 200 OK
real	0m11.395s
HTTP/1.1 200 OK
real	0m11.906s
HTTP/1.1 200 OK
real	0m12.079s
...
HTTP/1.1 200 OK
real	0m11.297s

So with the increased requests, the page load times increase from 2 seconds all the way up to 11-12 seconds!

Determining why this happens will involve investigation outside the scope of this article. However at least now we know which site doesn’t perform well under increased requests.

While every site is different, lets say www.example2.com was a WordPress site. Try installing the WordPress W3 Total Cache (W3TC) plugin. Basic items to enable/disable within W3TC would be:

- Enable Page Cache
- Disable Minify options
- Enable Browser Cache

Next, sign up with a free CloudFlare account, then add the following PageRules so CloudFlare actually starts caching your WordPress site:

1. *example2.com/wp-admin* : Cache Level (Bypass)
2. *example2.com/wp-login.php* : Cache Level (Bypass)
3. *example2.com/* : Cache Level (Cache Everything)
* The order is extremely critical here.  Make sure the pages you do not wish to cache are above the last line!
** Every site is different.  These rules may or may not work for you.

Then retest to see if your changes helped improve the times.

[user@workstation01 ~]# for i in {1..25}; do (time curl -sIL http://www.example2.com -X GET &) 2>&1 | grep -E "real|HTTP" & done

HTTP/1.1 200 OK
real	0m1.347s
HTTP/1.1 200 OK
real	0m1.342s
HTTP/1.1 200 OK
real	0m1.237s
HTTP/1.1 200 OK
real	0m1.021s
...
HTTP/1.1 200 OK
real	0m1.532s