Backing up permissions on directory

Before doing anything in Linux, it is also smart to have a rollback plan. Making blanket, recursive permission changes on a directory would certainly fall into this category!

Lets say you found a directory on your system where the file permissions were all 777, so you want to secure them a bit by changing the permissions over to 644. Something like:

[root@web01 ~]# find /var/www/vhosts/domain.com -type f -perm 0777 -print -exec chmod 644 {} \;

The paranoid among us will want to ensure we can revert things back to the way they were before. Thankfully there are two commands that can be used to either backup or restore permissions on a directory recursively: getfacl and setfacl

To backup all the permissions and ownerships within a given directory such as /var/www/vhosts/domain.com, do the following:

[root@web01 ~]# cd /var/www/vhosts/domain.com
[root@web01 ~]# getfacl -R . > permissions_backup

Now lets say you ran the find command, changed everything over to 644, then realized you broke your application cause it needed some files to be 664 or something, so you just want to roll back so you can investigate what happened.

You can roll back the permissions by running:

[root@web01 ~]# cd /var/www/vhosts/domain.com
[root@web01 ~]# setfacl --restore=permissions_backup

Backup entire servers permissions

If you wanted to backup the entire server’s permissions, you can do that by:

[root@web01 ~]# getfacl -R --absolute-names / > server_permissions_backup

And the restoration process remains the same:

[root@web01 ~]# setfacl --restore=server_permissions_backup

Find command examples

This is just a quick reference page for using find to do basic things.

Find is a pretty powerful tool that accepts a bunch of options for narrowing down your search. Some basic examples of stuff you can do are below.

Find a specific file type and extension older than 300 days and remove them

This will find files:
– Older than 300 days
– Is a file
– Match *.jpg
– Will not go into sub directories

This also works for those pesky directories that have millions of files.

First, always confirm the command will work before blindly removing files:

[root@web01 ~]# cd /path/to/directory
[root@web01 ~]# find . -maxdepth 1 -type f -name '*.jpg' -mtime +300 | xargs ls -al

Once you verified that the files displayed are the ones you want removed, remove them by running:

[root@web01 ~]# cd /path/to/directory
[root@web01 ~]# find . -maxdepth 1 -type f -name '*.jpg' -mtime +300 | xargs rm -f

Find files with 777 permissions

This will find all files that have 777 permissions:

[root@web01 ~]# cd /path/to/directory
[root@web01 ~]# find . -type f -perm 0777 -print

This will find all files that do NOT have 777 permissions

[root@web01 ~]# cd /path/to/directory
[root@web01 ~]# find / -type f ! -perm 777

Find Files with 777 Permissions and change to 644

Use caution with this, this is generally not smart to run blindly as it will go into subdirectories unless you set maxdepth.

[root@web01 ~]# cd /path/to/directory
[root@web01 ~]# find . -type f -perm 0777 -print -exec chmod 644 {} \;

Find Directories with 777 Permissions and change to 755

Use caution with this, this is generally not smart to run blindly as it will go into subdirectories unless you set maxdepth.

[root@web01 ~]# cd /path/to/directory
[root@web01 ~]# find . -type d -perm 777 -print -exec chmod 755 {} \;

Find empty directories

[root@web01 ~]# cd /path/to/directory
[root@web01 ~]# find /tmp -type d -empty

Find all hidden files within a directory

[root@web01 ~]# find /path/to/directory -type f -name ".*"

Find files owned by user or group

[root@web01 ~]# cd /path/to/directory
[root@web01 ~]# find /var/www -user apache
[root@web01 ~]# find /var/www -group apache

Find files that were modified in the last 30 days

[root@web01 ~]# find / -mtime 30

Find files that were modified in the last hour

[root@web01 ~]# find / -mmin -60

Find files that were changed within the last hour
Note, this one is specified in minutes only!

[root@web01 ~]# find / -cmin -60

Find files that were accessed in the last 5 days

[root@web01 ~]# find / -atime 5

Find files that were accessed within the last hour
Note, this one is specified in minutes only!

[root@web01 ~]# find / -amin -60

Count files per directory with find
This one is useful when you need to find the top 10 directories that contain the most amount of files.

[root@web01 ~]# vim count-files-per-directory.sh
#!/bin/bash

if [ $# -ne 1 ];then
  echo "Usage: `basename $0` DIRECTORY"
  exit 1
fi

echo "Please wait..."

find "$@" -type d -print0 2>/dev/null | while IFS= read -r -d '' file; do 
    echo -e `ls -A "$file" 2>/dev/null | wc -l` "files in:\t $file"
done | sort -nr | head | awk '{print NR".", "\t", $0}'

exit 0

Now run it against the / directory:

[root@web01 ~]# bash count-files-per-directory.sh /
Please wait...
1. 	 768 files in:	 /usr/share/man/man1
2. 	 631 files in:	 /usr/lib64/python2.6
3. 	 575 files in:	 /usr/share/locale
4. 	 566 files in:	 /usr/share/vim/vim74/syntax
5. 	 496 files in:	 /usr/bin
6. 	 487 files in:	 /usr/share/man/man8
7. 	 393 files in:	 /usr/share/perl5/unicore/lib/gc_sc
8. 	 380 files in:	 /usr/include/linux
9. 	 354 files in:	 /usr/lib64/python2.6/encodings
10. 	 334 files in:	 /usr/share/man/man3

Or if you only need to run the search in a specific directory:

[root@web01 ~]# bash count-files-per-directory.sh /usr/share/man
Please wait...
1. 	 768 files in:	 /usr/share/man/man1
2. 	 487 files in:	 /usr/share/man/man8
3. 	 334 files in:	 /usr/share/man/man3
4. 	 124 files in:	 /usr/share/man/man5
5. 	 49 files in:	 /usr/share/man
6. 	 35 files in:	 /usr/share/man/ru/man8
7. 	 31 files in:	 /usr/share/man/man7
8. 	 27 files in:	 /usr/share/man/fr/man8
9. 	 25 files in:	 /usr/share/man/de/man8
10. 	 22 files in:	 /usr/share/man/ja/man8

Rolling back yum transactions

Ever had the system update a package, which winds up breaking the most random things? How can you roll back? How can you prevent that same buggy package from updating itself again the next time the system checks for updates, yet still get newer versions of that package when its released?

I ran across something like this recently. The symptom was that PHPMyAdmin was no longer working on this LAMP server. In short, it was found that an Apache update was to blame, which was found in this bug report: https://bz.apache.org/bugzilla/show_bug.cgi?id=61202

So how can the update to Apache be rolled back? First, try to confirm that Apache was indeed updated recently:

[root@web01 ~]# tail /var/log/yum.log
Jul 08 04:23:49 Updated: httpd24u-filesystem-2.4.26-1.ius.centos6.noarch
Jul 08 04:23:49 Updated: httpd24u-tools-2.4.26-1.ius.centos6.x86_64
Jul 08 04:23:50 Updated: httpd24u-2.4.26-1.ius.centos6.x86_64
Jul 08 04:23:50 Updated: 1:httpd24u-mod_ssl-2.4.26-1.ius.centos6.x86_64

Now find the transaction ID within yum by running:

[root@web01 ~]# yum history
ID     | Login user               | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
   220 | root               | 2017-07-08 04:23 | Update         |    4

View the details of this transaction by running:

[root@web01 ~]# yum history info 220
...
Transaction performed with:
    Installed     rpm-4.8.0-55.el6.x86_64                       @centos6-x86_64
    Installed     yum-3.2.29-81.el6.centos.noarch               @centos6-x86_64
    Installed     yum-metadata-parser-1.1.2-16.el6.x86_64       @anaconda-CentOS-201410241409.x86_64/6.6
    Installed     yum-plugin-fastestmirror-1.1.30-40.el6.noarch @centos6-x86_64
    Installed     yum-rhn-plugin-2.4.6-1.el6.noarch             @spacewalk
Packages Altered:
    Updated httpd24u-2.4.25-4.ius.centos6.x86_64            @rackspace-centos6-x86_64-ius
    Update           2.4.26-1.ius.centos6.x86_64            @rackspace-centos6-x86_64-ius
    Updated httpd24u-filesystem-2.4.25-4.ius.centos6.noarch @rackspace-centos6-x86_64-ius
    Update                      2.4.26-1.ius.centos6.noarch @rackspace-centos6-x86_64-ius
    Updated httpd24u-mod_ssl-1:2.4.25-4.ius.centos6.x86_64  @rackspace-centos6-x86_64-ius
    Update                   1:2.4.26-1.ius.centos6.x86_64  @rackspace-centos6-x86_64-ius
    Updated httpd24u-tools-2.4.25-4.ius.centos6.x86_64      @rackspace-centos6-x86_64-ius
    Update                 2.4.26-1.ius.centos6.x86_64      @rackspace-centos6-x86_64-ius
history info
...

To roll back the updates, getting us back to Apache 2.4.25 in this case, simple undo the transaction by running:

[root@web01 ~]# yum history undo 220

Then confirm Apache is back to the previous version 2.4.25:

[root@web01 ~]# rpm -qa |grep -i httpd24u
httpd24u-filesystem-2.4.25-4.ius.centos6.noarch
httpd24u-2.4.25-4.ius.centos6.x86_64
httpd24u-mod_ssl-2.4.25-4.ius.centos6.x86_64
httpd24u-tools-2.4.25-4.ius.centos6.x86_64

Next, restart Apache so the changes take place:

[root@web01 ~]# service httpd restart

Finally, exclude the buggy packages from ever being installed again. In this example, Apache 2.4.26 will never be installed, however any newer versions released after that will install/update normally.

[root@web01 ~]# yum install yum-plugin-versionlock
[root@web01 ~]# yum versionlock add! httpd24u-mod_ssl-2.4.26-1.ius.centos6.x86_64 httpd24u-2.4.26-1.ius.centos6.x86_64 httpd24u-tools-2.4.26-1.ius.centos6.x86_64 httpd24u-filesystem-2.4.26-1.ius.centos6.noarch

Setting default kernel in grub2

With newer systems like CentOS 7 and Ubuntu 14.04 and 16.04 using Grub2, you can no longer simply update a single file to have your kernel boot off an older or newer kernel. There are a series of steps that must be followed. The examples below will show how to boot off an older kernel for their respective operating systems.

Please note, the instructions in this article will lock your kernel on whichever one you selected. Even if your system receives automatic kernel updates, those new kernels will have to be manually enabled within grub if you want to use them.

CentOS 7

First, check to see which kernel is currently running:

[root@web01 ~]# uname -r
3.10.0-514.16.1.el7.x86_64

That shows us we’re running 3.10.0-514.16.1, however I need to be running 3.10.0-327.36.3. So to use this specific named kernel, first changed the GRUB_DEFAULT to ‘saved’ in /etc/default/grub by:

[root@web01~]# cp /etc/default/grub /etc/default/grub.bak
[root@web01~]# vim /etc/default/grub
...
GRUB_DEFAULT=saved
...

Now create a backup of the grub config for recovery purposes if needed, then rebuild grub:

[root@web01 ~]# cp /boot/grub2/grub.cfg /boot/grub2/grub.cfg.bak
[root@web01 ~]# grub2-mkconfig -o /boot/grub2/grub.cfg

Determine what the full kernel name is you want to use. Get the listing by running:

[root@web01~]# grep "^menuentry" /boot/grub2/grub.cfg | cut -d "'" -f2
CentOS Linux (3.10.0-514.16.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-514.2.2.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-327.36.3.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-327.22.2.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-327.3.1.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-c11d017c89ca4e8685ae3d9791d472ca) 7 (Core)

I want to use the 3.10.0-327.36.3 kernel. Setting that to be the default kernel is simple. Set the desired kernel by running:

[root@web01 ~]# grub2-set-default "CentOS Linux (3.10.0-327.36.3.el7.x86_64) 7 (Core)"

Now verify that the change got applied in the configs by running:

[root@web01 ~]# grub2-editenv list
saved_entry=CentOS Linux (3.10.0-327.36.3.el7.x86_64) 7 (Core)

Reboot the system so it boots off the older kernel:

root@web01:~# reboot

Finally, once the system comes back online, verify the desired kernel is running by:

[root@web01 ~]# uname -r
3.10.0-327.36.3.el7.x86_64

If the system rebooted, and dropped you into a grub shell with an error, you can boot up off of the backup grub.cfg file that was created by:

grub2> configfile (hd0,1)/boot/grub2/grub.cfg.bak

Ubuntu 14.04 and Ubuntu 16.04

First, check to see which kernel is currently running:

root@web01:~# uname -r
4.4.0-48-generic

That shows us we’re running 4.4.0-48, however I need to be running 4.4.0-47. So to use this specific named kernel, first changed the GRUB_DEFAULT to ‘saved’ in /etc/default/grub by:

root@web01:~# cp /etc/default/grub /etc/default/grub.bak
root@web01:~# vim /etc/default/grub
...
GRUB_DEFAULT=saved
...

Now create a backup of the grub config for recovery purposes if needed, then rebuild grub:

root@web01:~# cp /boot/grub/grub.cfg /boot/grub/grub.cfg.bak
root@web01:~# update-grub

Determine what the full kernel name is you want to use. Get the listing by running:

root@web01:~# egrep "^[[:space:]]?(submenu|menuentry)" /boot/grub/grub.cfg | cut -d "'" -f2
Ubuntu
Advanced options for Ubuntu
Ubuntu, with Linux 4.4.0-78-generic
Ubuntu, with Linux 4.4.0-78-generic (recovery mode)
Ubuntu, with Linux 4.4.0-47-generic
Ubuntu, with Linux 4.4.0-47-generic (recovery mode)

I want to use the 4.4.0-47-generic kernel. Setting that to be the default kernel is simple. However, you MUST prepend ‘Advanced options for Ubuntu’ to the kernel name as shown below since Ubuntu makes use of sub menus in the kernel listing. So set the desired kernel by running:

root@web01:~# grub-set-default 'Advanced options for Ubuntu>Ubuntu, with Linux 4.4.0-47-generic'

Now verify that the change got applied in the configs by running:

root@web01:~# grub-editenv list
saved_entry=Advanced options for Ubuntu>Ubuntu, with Linux 4.4.0-47-generic

Reboot the system so it boots off the older kernel:

root@web01:~# reboot

Finally, once the system comes back online, verify the desired kernel is running by:

root@web01:~# uname -r
4.4.0-47-generic

If the system rebooted, and dropped you into a grub shell with an error, you can boot up off of the backup grub.cfg file that was created by:

grub2> configfile (hd0,1)/boot/grub2/grub.cfg.bak

Setting up the old-releases repo for ubuntu

This is a guide on enabling the old-releases repos for the Ubuntu project. When Ubuntu marks a release EOL, the mainline repos are moved to an alternate location where they are preserved for historical purposes. It should be noted that when a system reaches EOL, the repos for it are no longer maintained.

It goes without saying that enabling the old-releases repos for EOL systems should only be used as a last resort. Continuing to run a system that has reached the end of life status is dangerous as it no longer receives security patches and bug fixes. There are also no promises that things will continue to work.

Mitigating security and reliability issues can be resolved all together by updating the system to a supported version of the operating system. If there is a compelling reason to enable the old-releases repos so packages can be installed, then proceed below.

Ubuntu 10.04 EOL Repos

Ubuntu 10.04 LTS went EOL on 4/2015. The procedure for setting up the system to use the old-releases repos are below.

First, create a backup of the /etc/apt/sources.list by:

[root@web01 ~]# cp /etc/apt/sources.list /etc/apt/sources.list.bak

Now update /etc/apt/sources.list to point to the old-releases repos accordingly. Keep in mind that there may be repos specified in here for Nginx, Varnish, Docker, etc. So be sure to only update the items needed by Ubuntu. The end result should look something like this:

[root@web01 ~]# vim /etc/apt/sources.list
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.

deb http://old-releases.ubuntu.com/ubuntu/ lucid main restricted
deb-src http://old-releases.ubuntu.com/ubuntu/ lucid main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://old-releases.ubuntu.com/ubuntu/ lucid-updates main restricted
deb-src http://old-releases.ubuntu.com/ubuntu/ lucid-updates main restricted

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://old-releases.ubuntu.com/ubuntu/ lucid universe
deb-src http://old-releases.ubuntu.com/ubuntu/ lucid universe
deb http://old-releases.ubuntu.com/ubuntu/ lucid-updates universe
deb-src http://old-releases.ubuntu.com/ubuntu/ lucid-updates universe

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu 
## team, and may not be under a free licence. Please satisfy yourself as to 
## your rights to use the software. Also, please note that software in 
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://old-releases.ubuntu.com/ubuntu/ lucid multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ lucid multiverse
deb http://old-releases.ubuntu.com/ubuntu/ lucid-updates multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ lucid-updates multiverse

## Uncomment the following two lines to add software from the 'backports'
## repository.
## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
# deb http://old-releases.ubuntu.com/ubuntu/ lucid-backports main restricted universe multiverse
# deb-src http://old-releases.ubuntu.com/ubuntu/ lucid-backports main restricted universe multiverse

## Uncomment the following two lines to add software from Canonical's
## 'partner' repository.
## This software is not part of Ubuntu, but is offered by Canonical and the
## respective vendors as a service to Ubuntu users.
# deb http://old-releases.ubuntu.com/ubuntu lucid partner
# deb-src http://old-releases.ubuntu.com/ubuntu lucid partner

deb http://old-releases.ubuntu.com/ubuntu lucid-security main restricted
deb-src http://old-releases.ubuntu.com/ubuntu lucid-security main restricted
deb http://old-releases.ubuntu.com/ubuntu lucid-security universe
deb-src http://old-releases.ubuntu.com/ubuntu lucid-security universe
deb http://old-releases.ubuntu.com/ubuntu lucid-security multiverse
deb-src http://old-releases.ubuntu.com/ubuntu lucid-security multiverse

Now refresh the package index from their sources by running:

[root@web01 ~]# apt-get update

Address any 404’s accordingly as that means the URL may be incorrect or may not longer exist.

Ubuntu 12.04 EOL Repos

Ubuntu 12.04 LTS is going EOL on 4/2017. The procedure for setting up the system to use the old-releases repos are below.

First, create a backup of the /etc/apt/sources.list by:

[root@web01 ~]# cp /etc/apt/sources.list /etc/apt/sources.list.bak

Now update /etc/apt/sources.list to point to the old-releases repos accordingly. Keep in mind that there may be repos specified in here for Nginx, Varnish, Docker, etc. So be sure to only update the items needed by Ubuntu. The end result should look something like this:

[root@web01 ~]# vim /etc/apt/sources.list
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://old-releases.ubuntu.com/ubuntu/ precise main restricted
deb-src http://old-releases.ubuntu.com/ubuntu/ precise main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://old-releases.ubuntu.com/ubuntu/ precise-updates main restricted
deb-src http://old-releases.ubuntu.com/ubuntu/ precise-updates main restricted

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://old-releases.ubuntu.com/ubuntu/ precise universe
deb-src http://old-releases.ubuntu.com/ubuntu/ precise universe
deb http://old-releases.ubuntu.com/ubuntu/ precise-updates universe
deb-src http://old-releases.ubuntu.com/ubuntu/ precise-updates universe

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu 
## team, and may not be under a free licence. Please satisfy yourself as to 
## your rights to use the software. Also, please note that software in 
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://old-releases.ubuntu.com/ubuntu/ precise multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ precise multiverse
deb http://old-releases.ubuntu.com/ubuntu/ precise-updates multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ precise-updates multiverse

## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
deb http://old-releases.ubuntu.com/ubuntu/ precise-backports main restricted universe multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ precise-backports main restricted universe multiverse

deb http://old-releases.ubuntu.com/ubuntu precise-security main restricted
deb-src http://old-releases.ubuntu.com/ubuntu precise-security main restricted
deb http://old-releases.ubuntu.com/ubuntu precise-security universe
deb-src http://old-releases.ubuntu.com/ubuntu precise-security universe
deb http://old-releases.ubuntu.com/ubuntu precise-security multiverse
deb-src http://old-releases.ubuntu.com/ubuntu precise-security multiverse

## Uncomment the following two lines to add software from Canonical's
## 'partner' repository.
## This software is not part of Ubuntu, but is offered by Canonical and the
## respective vendors as a service to Ubuntu users.
# deb http://old-releases.ubuntu.com/ubuntu precise partner
# deb-src http://old-releases.ubuntu.com/ubuntu precise partner

## Uncomment the following two lines to add software from Ubuntu's
## 'extras' repository.
## This software is not part of Ubuntu, but is offered by third-party
## developers who want to ship their latest software.
# deb http://old-releases.ubuntu.com/ubuntu precise main
# deb-src http://old-releases.ubuntu.com/ubuntu precise main

Now refresh the package index from their sources by running:

[root@web01 ~]# apt-get update

Address any 404’s accordingly as that means the URL may be incorrect or may not longer exist.

Using hpasmcli for HP servers

HP comes with their server utility scripts called hpssacli and hpacucli. These tools allow you to view and modify your hardware configuration on the server. The hpacucli is the older implementation of the toolkit, but the syntax is pretty similar.

HP tools information

To show the firmware version, run:

[root@web01 ~]# hpasmcli -s "show server"

If you want to see extended information, run:

[root@web01 ~]# hpssacli controller all show config detail

General information

To view information regarding the server model, cpu, type, memory, etc, run:

[root@web01 ~]# hpasmcli -s "show server"

Hardware Health

If you want to view the health of the system and chassis components, run:

[root@web01 ~]# hpasmcli -s "show server"

The chassis can also only return specific components, such as:

[root@web01 ~]# hpasmcli -s "show powersupply"
[root@web01 ~]# hpasmcli -s "show dimm"
[root@web01 ~]# hpasmcli -s "show fans"
[root@web01 ~]# hpasmcli -s "show temp"

Storage health

To view the physical and virtual disks on the server:

[root@web01 ~]# hpssacli ctrl all show config
[root@web01 ~]# hpssacli controller slot=3 physicaldrive all show
[root@web01 ~]# hpssacli controller slot=3 physicaldrive 2I:1:5 show detail
[root@web01 ~]# hpssacli controller slot=3 logicaldrive all show

On older HP servers, you can view the physical and virtual disks on the server by:

[root@web01 ~]# hpacucli controller slot=1 physicaldrive all show
[root@web01 ~]# hpacucli controller slot=1 physicaldrive 2I:1:5 show detail
[root@web01 ~]# hpacucli controller slot=1 logicaldrive all show

To see the storage battery status:

[root@web01 ~]# hpssacli controller all show detail | egrep -i 'battery\/|controller\ status|cache\ status'
   Controller Status: OK
   Cache Status: OK
   Battery/Capacitor Count: 1
   Battery/Capacitor Status: OK

Hardware logs

To display the hardware logs:

[root@web01 ~]# hpasmcli -s "show iml"

If you need to clear the hardware logs:

[root@web01 ~]# hpasmcli -s "clear iml"

CPU actions

To see if hyperthreading is enabled on the CPUs:

[root@web01 ~]# hpasmcli -s "show ht"

If you wanted to change the hyperthreading settings:

# Enable
[root@web01 ~]# hpasmcli -s "enable ht"

# Disable
[root@web01 ~]# hpasmcli -s "disable ht"

Using omreport and omconfig for Dell servers

Dell comes with their server utility scripts called omreport and omconfig. These tools allow you to view and modify your hardware configuration on the server.

Dell tools information

To see what version of the tools your running:

[root@web01 ~]# omreport about details=true

To see if there are updates available for the firmware:

[root@web01 ~]# omreport system version

To see what commands are available using omreport:

[root@web01 ~]# omreport system -?

General information

To view information regarding the server model, cpu type, memory, service tags, etc, run:

[root@web01 ~]# omreport system summary

Hardware Health

If you want to view the health of the system and chassis components, run:

[root@web01 ~]# omreport system

To only get the health information for the chassis:

[root@web01 ~]# omreport chassis

The chassis can also only return specific components, such as:

[root@web01 ~]# omreport chassis fans
[root@web01 ~]# omreport chassis memory
[root@web01 ~]# omreport chassis nics
[root@web01 ~]# omreport chassis processors
[root@web01 ~]# omreport chassis temps
[root@web01 ~]# omreport chassis batteries
[root@web01 ~]# omreport chassis pwrsupplies

Storage health

As a quick note, if the commands below report there are no controllers listed, check to be sure that the software is actually running by:

[root@web01 ~]# /opt/dell/srvadmin/sbin/srvadmin-services.sh status
dell_rbu (module) is stopped
ipmi driver is running
dsm_sa_datamgrd is stopped
dsm_sa_eventmgrd is stopped
dsm_sa_snmpd is stopped
dsm_om_shrsvcd is stopped
dsm_om_connsvcd is stopped
[root@web01 ~]# /opt/dell/srvadmin/sbin/srvadmin-services.sh restart

To view the physical and virtual disks on the server:

[root@web01 ~]# omreport storage pdisk controller=0
[root@web01 ~]# omreport storage vdisk controller=0
[root@web01 ~]# omreport storage pdisk controller=0 vdisk=0

If you just wanted a quick listing of the relevant disk information to see the state of the drives, run:

[root@web01 ~]# omreport storage pdisk controller=0 | grep -iE "^id|^status|name|state|Failure Predicted"
ID                              : 0:0:0
Status                          : Ok
Name                            : Physical Disk 0:0:0
State                           : Online
Failure Predicted               : No
ID                              : 0:0:1
Status                          : Ok
Name                            : Physical Disk 0:0:1
State                           : Online
Failure Predicted               : No

To see if there are any empty drive bays on the server:

[root@web01 ~]# omreport storage controller controller=0 info=pdslotreport | grep 'Empty Slots'

To see the storage battery status:

[root@web01 ~]# omreport storage battery controller=0

Hardware Logs

To display the hardware logs, run:

[root@web01 ~]# omreport system esmlog

If you need to view the alert logs:

[root@web01 ~]# omreport system alertlog

And if you needed to view the messages from the POST:

[root@web01 ~]# omreport system postlog

If you find you need to clear the logs, that can be performed by:

[root@web01 ~]# omconfig system esmlog action=clear
[root@web01 ~]# omconfig system alertlog action=clear
[root@web01 ~]# omconfig system postlog action=clear

CPU actions

To see if hyperthreading is enabled on the CPUs:

[root@web01 ~]# omreport chassis biossetup | grep -A 2 'HyperThreading'

If you wanted to enable hyperthreading:

# Dell R710
[root@web01 ~]# omconfig chassis biossetup attribute=cpuht setting=enabled

# Dell R720
[root@web01 ~]# omconfig chassis biossetup attribute=ProcCores setting=All

If you needed to enable or disable NUMA:

# Disable NUMA:
[root@web01 ~]# omconfig chassis biossetup attribute=numa setting=disabled

# Enable NUMA:
[root@web01 ~]# omconfig chassis biossetup attribute=numa setting=enabled

Apache quick stats

When troubleshooting performance issues with Apache that happened earlier in the day or week, it is very useful to parse the logs quickly to determine quick facts about the inbound connection rates. It may reveal a period of increased traffic that needs to be investigated further, therefore giving you that thread to begin unraveling the problem.

To get the total connections per day for a website, run the following:

[root@web01 ~]# cat /var/log/httpd/www.example.com-access.log | awk '{print $4}' | cut -d: -f1 |uniq -c
   1247345 [20/Feb/2017
   1331908 [21/Feb/2017
   1295677 [22/Feb/2017
   1435275 [23/Feb/2017
   1023423 [24/Feb/2017
   1342332 [25/Feb/2017
   1293422 [26/Feb/2017
   2131198 [27/Feb/2017

To get the total connections per day for each website on the server, run the following:

[root@web01 ~]# for i in `ls /var/log/httpd/*-access.log`; do echo $i && cat $i | awk '{print $4}' | cut -d: -f1 |uniq -c && echo ""; done
/var/log/httpd/www.example.com-access.log
   1247345 [20/Feb/2017
   1331908 [21/Feb/2017
   1295677 [22/Feb/2017
   1435275 [23/Feb/2017
   1023423 [24/Feb/2017
   1342332 [25/Feb/2017
   1293422 [26/Feb/2017
   2131198 [27/Feb/2017

/var/log/httpd/www.example02.com-access.log
   2542 [20/Feb/2017
   7586 [21/Feb/2017
   4776 [22/Feb/2017
   2975 [23/Feb/2017
  16756 [24/Feb/2017
   9874 [25/Feb/2017
   1638 [26/Feb/2017
   9654 [27/Feb/2017

To get the connections per hour for a specific day, run the following:

[root@web01 ~]# grep "27/Feb" /var/log/httpd/www.example.com-access.log | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":00"}' | sort -n | uniq -c
  50205 03:00
  90516 04:00
  64837 05:00
  47410 06:00
  44876 07:00
  41098 08:00
  38996 09:00
  37234 10:00
  43704 11:00
  58702 12:00
  58922 13:00
  72592 14:00
  77792 15:00
  88882 16:00
  80815 17:00
  98287 18:00
 617857 19:00
  90507 20:00
  98568 21:00
 147584 22:00
 181814 23:00

Based off that output, there was a massive spike in connections during the 9:00PM hour (19:00). So now lets break the 9:00PM hour down to show the connections per minute:

[root@web01 ~]# grep "27/Feb/2017:19" /var/log/httpd/www.example.com-access.log | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":"$3}' | sort -nk1 -nk2 | uniq -c | awk '{ if ($1 > 10) print $0}'
   1629 19:00
   1664 19:01
   1840 19:02
  10493 19:03
  13728 19:04
  17608 19:05
   1377 19:06
   2333 19:07
   1980 19:08
   2056 19:09
   2123 19:10
...
   1997 19:57
   1631 19:58
   1988 19:59

As shown above, there was some sort of traffic spike that occurred between 9:03PM – 9:05PM. As the window has been narrowed down to a 3 minute period, more specific analysis can be performed. The examples below will focus on what was happening around 9:03PM.

To list the top 10 IP’s accessing the site during around 9:03PM

[root@web01 ~]# grep "27/Feb/2017:19:03" /var/log/httpd/www.example.com-access.log | awk '{print $1}' | sort -nr | uniq -c |sort -nr | head

To list the top most called elements on the site:

[root@web01 ~]# grep "27/Feb/2017:19:03" /var/log/httpd/www.example.com-access.log | awk '{print $7}' | sort -nr | uniq -c | sort -nr | head

To show the bandwidth for a domain use the command below:

# Daily bandwidth total
[root@web01 ~]# grep '27/Feb/2017:' /var/log/httpd/www.example.com-access.log | grep -oP 'HTTP/1.[01]" [0-9]{3} [0-9]+' | awk '{SUM+=$3} END { print SUM / 1024 / 1024 / 1024 " GB" }'

# Monthly bandwidth total
[root@web01 ~]# grep '/Feb/2017:' /var/log/httpd/www.example.com-access.log | grep -oP 'HTTP/1.[01]" [0-9]{3} [0-9]+' | awk '{SUM+=$3} END { print SUM / 1024 / 1024 / 1024 " GB" }'

To get a count of status codes to identify any trends:

# Get all status codes
[root@web01 ~]# cat /var/log/httpd/www.example.com-access.log |awk '{print $9}' | sort -nr | uniq -c |sort -nr
  36355 200
   4896 304
   3942 404
   1599 302
    301 301
    195 403
      4 400
      3 401

# Get summary of top 10 404's:
[root@web01 ~]# awk '($9 ~ /404/)' /var/log/httpd/www.example.com-access.log | awk '{print $9,$7}' | sort -nr | uniq -c |sort -nr | head
   1369 404 /apple-touch-icon-precomposed.png
   1369 404 /apple-touch-icon.png
    502 404 /apple-touch-icon-120x120-precomposed.png
    502 404 /apple-touch-icon-120x120.png
     22 404 /apple-touch-icon-152x152-precomposed.png
     22 404 /apple-touch-icon-152x152.png
     21 404 /news/html
      5 404 /components/com_foxcontact/lib/file-uploader.php
      3 404 /blog/wp-login.php
      1 404 /author/wp-login.php

Changing your servers timezone

The systems timezone is usually set during installation. If the timezone needed to be changed, it can be done without rebooting the system.

Just be sure to watch for applications like MySQL and PHP that require additional steps for changing the timezone, which are noted near the bottom of this article.

CentOS 5 and CentOS 6

Modify the zone in /etc/sysconfig/clock. You can find the valid timezones in /usr/share/zoneinfo. The commonly used timezones include America/New_York, America/Chicago, America/Los_Angeles, and UTC.

[root@web01 ~]# vim /etc/sysconfig/clock
...
ZONE="America/New_York"
...

Then update /etc/localtime:

[root@web01 ~]# tzdata-update

Now sync the hardware clock against the system time:

[root@web01 ~]# hwclock --systohc

Go ahead and restart syslogd/rsyslogd and crond:

[root@web01 ~]# service crond restart
[root@web01 ~]# service rsyslog restart
[root@web01 ~]# service syslog restart

CentOS 7

Changing the timezone on CentOS 7 can be done with a few commands. You can find the valid timezones in /usr/share/zoneinfo. The commonly used timezones include America/New_York, America/Chicago, America/Los_Angeles, and UTC.

[root@web01 ~]# timedatectl set-timezone America/New_York
[root@web01 ~]# systemctl restart crond
[root@web01 ~]# systemctl restart rsyslog

Ubuntu 12.04 and Ubuntu 14.04

Modify the zone in /etc/timezone. You can find the valid timezones in /usr/share/zoneinfo. The commonly used timezones include America/New_York, America/Chicago, America/Los_Angeles, and UTC.

[root@web01 ~]# vim /etc/timezone
...
America/New_York
...

Now update active timezone:

[root@web01 ~]# dpkg-reconfigure --frontend noninteractive tzdata
Current default time zone: 'America/New_York'
Local time is now:      Tue Jan 17 01:18:04 EST 2017.
Universal Time is now:  Tue Jan 17 06:18:04 UTC 2017.

Restart rsyslog and cron:

[root@web01 ~]# service cron restart
[root@web01 ~]# service rsyslog restart

Ubuntu 16.04

Changing the timezone on Ubuntu 16.04 can be done with a few commands. You can find the valid timezones in /usr/share/zoneinfo. The commonly used timezones include America/New_York, America/Chicago, America/Los_Angeles, and UTC.

[root@web01 ~]# timedatectl set-timezone America/New_York
[root@web01 ~]# systemctl restart crond
[root@web01 ~]# systemctl restart rsyslog

MySQL, Percona and MariaDB

In order for MySQL, Percona, and MariaDB to register the new timezone settings, they need to be restarted. There really isn’t a way around this. As a temporary solution, and one that will not pick up future DST timezone changes, you can manually set the databases current time by:

# List current system date and time via MySQL:
[root@web01 ~]# mysql
mysql> SELECT @@global.time_zone;

# List the current date and time according to MySQL:
mysql> SELECT NOW();

# Update the timezone using the UTC offset:
mysql> SET @@global.time_zone = '+05:00';

Even with this temporary fix in place, unless you are using the UTC timezone, MySQL should be restarted very soon.

PHP

PHP should also have its timezone updated when you change it on the system. Determine where your php.ini resides, then update it accordingly. I am assuming CentOS 6 for this example:

[root@web01 ~]# vim /etc/php.ini
...
date.timezone = "America/New_York"
...

Then restart Apache or PHP-FPM for your system so the changes are applied. Afterwards, test the timezone change to PHP by:

[root@web01 ~]# php -i |grep date.timezone
date/time support => enabled
date.timezone => America/New_York => America/New_York

A list of supported timezone configurations for PHP can be found at:
http://www.php.net/manual/en/timezones.php

Changing the servers hostname

The hostname of a system is usually set during installation. However as time goes on, it may be determined that the hostname should be changed to something more suitable.

Outlined below is the procedure for changing the systems hostname without having to reboot.

CentOS 5 and 6

First, check to see what the existing hostname is on the server:

[root@web01 ~]# hostname

Then change the current hostname. In this example, we’re going to change the hostname to web04.mydomain.com:

[root@web01 ~]# hostname web04.mydomain.com

Now update /etc/hosts with the new hostname:

[root@web01 ~]# vim /etc/hosts
127.0.0.1      localhost.localdomain.com localhost
192.168.1.5    web04.mydomain.com web04

If you have ‘domain’ specified in /etc/resolv.conf, update that accordingly. There is not a need to add ‘domain’ if it is not already defined:

[root@web01 ~]# vim /etc/resolv.conf
domain web04.mydomain.com
nameserver 8.8.8.8
nameserver 8.8.4.4

Next update your network configuration:

[root@web01 ~]# vim /etc/sysconfig/network
...
HOSTNAME=web04.mydomain.com
...

Restart syslog so the new changes go into effect:

# CentOS 5
[root@web01 ~]# service syslog restart

# CentOS 6
[root@web01 ~]# service rsyslog restart

Finally, log out and back into the server via SSH and you should see the new hostname in effect. Also check /var/log/secure and /var/log/messages to ensure the new hostname is being used.

CentOS 7

In this example, we’re going to change the hostname to web04.mydomain.com. First run:

[root@web01 ~]# hostname web04.mydomain.com

Now update /etc/hosts with the new hostname:

[root@web01 ~]# vim /etc/hosts
127.0.0.1      localhost.localdomain.com localhost
192.168.1.5    web04.mydomain.com web04

Now update the hostname using the systemd command:

[root@web01 ~]# hostnamectl set-hostname web04.mydomain.com

If you have ‘domain’ specified in /etc/resolv.conf, update that accordingly. There is not a need to add ‘domain’ if it is not already defined:

[root@web01 ~]# vim /etc/resolv.conf
domain web04.mydomain.com
nameserver 8.8.8.8
nameserver 8.8.4.4

Now restart syslog so the new changes go into effect:

[root@web01 ~]# systemctl restart rsyslog

Finally, log out and back into the server via SSH and you should see the new hostname in effect. Also check /var/log/auth.log and /var/log/syslog to ensure the new hostname is being used.

Ubuntu 12.04 and Ubuntu 14.04

First, check to see what the existing hostname is on the server:

[root@web01 ~]# hostname

In this example, we’re going to change the hostname to web04.mydomain.com. So run:

[root@web01 ~]# hostname web04.mydomain.com

Now update /etc/hosts with the new hostname:

[root@web01 ~]# vim /etc/hosts
127.0.0.1      localhost.localdomain.com localhost
192.168.1.5    web04.mydomain.com web04

If you have ‘domain’ specified in /etc/resolv.conf, update that accordingly. There is not a need to add ‘domain’ if it is not already defined:

[root@web01 ~]# vim /etc/resolv.conf
domain web04.mydomain.com
nameserver 8.8.8.8
nameserver 8.8.4.4

Then update /etc/hostname accordingly:

[root@web01 ~]# vim /etc/hostname
web04.mydomain.com

Now restart syslog so the new changes go into effect:

[root@web01 ~]# service rsyslog restart

Finally, log out and back into the server via SSH and you should see the new hostname in effect. Also check /var/log/auth.log and /var/log/syslog to ensure the new hostname is being used.

Ubuntu 16.04 and Ubuntu 18.04

First, check to see what the existing hostname is on the server:

[root@web01 ~]# hostname

In this example, we’re going to change the hostname to web04.mydomain.com. So run:

[root@web01 ~]# hostname web04.mydomain.com

Now update /etc/hosts with the new hostname:

[root@web01 ~]# vim /etc/hosts
127.0.0.1      localhost.localdomain.com localhost
192.168.1.5    web04.mydomain.com web04

Now update the hostname using the systemd command:

[root@web01 ~]# hostnamectl set-hostname web04.mydomain.com

If you have ‘domain’ specified in /etc/resolv.conf, update that accordingly. There is not a need to add ‘domain’ if it is not already defined:

[root@web01 ~]# vim /etc/resolv.conf
domain web04.mydomain.com
nameserver 8.8.8.8
nameserver 8.8.4.4

Now restart syslog so the new changes go into effect:

[root@web01 ~]# systemctl restart rsyslog

On Ubuntu 18.04 only, check to see if /etc/cloud/cloud.cfg exists. If it does, confirm the preserve hostname is set to true as shown below:

[root@web01 ~]# vim /etc/cloud/cloud.cfg
...
preserve_hostname: true
...

Finally, log out and back into the server via SSH and you should see the new hostname in effect. Also check /var/log/auth.log and /var/log/syslog to ensure the new hostname is being used.