Tag: aide

Using AIDE for file integrity monitoring (FIM) on Ubuntu or Debian

PCI-DSS 3.1 section 10.5.5 has the following requirement:

Use file-integrity monitoring or change-detection software on logs to ensure that existing log data cannot be changed without generating alerts (although new data being added should not cause an alert).

For large solutions, I would suggest using a well known tool such as Tripwire Enterprise. However many small to mid size companies that have a small footprint within their card holder data environment (CDE), may not be able to afford this. So what can companies use to meet this requirement? Implement AIDE (Advanced Intrusion Detection Environment).

Taken from the projects website, AIDE creates a database from the regular expression rules that it finds from the config file(s). Once this database is initialized it can be used to verify the integrity of the files.

AIDE is a very simple (yet powerful) program that runs from cron checking your files (typically once a night), and it will scan your system looking for any changes in the directories its monitoring. There are a number of different ways to use this program, but I’ll outline one that I like to use.

My requirements:
1. I want the reports to run nightly.
2. All change reports are emailed to me so I can archive them for a year offsite.
3. Have the database automatically commit the additions, deletions, and changes to baseline each time its ran.

In the event my system was compromised, I want to ensure that the malicious user was not able to modify, or delete my previous reports. Therefore, I choose not to store them on the machine. Its true that once the malicious user gained access to my system, they could change my AIDE config on me, but at least my previous reports will be intact which should help me when determining what malicious changes this user made to my server. Please note that I am making an assumption here that you are already backing up your system nightly, which would include your AIDE database! If you do not currently have a backup strategy in place, get one. Tools such as AIDE helps identify what files a malicious user may have changed, but if they completely crippled the system, you will need to restore from backups.

Setting up AIDE is fairly straight forward. It exists in most of package repositories out there including most variants of Linux and BSD.

On Ubuntu or Debian based systems, you can install it by:

[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install aide

Now to setup some basic configurations, such as the email notifications, update type, etc, modify the AIDE system configuration file according:

[root@web01 ~]# vim /etc/default/aide
...
FQDN=web01.domain.com
MAILSUBJ="Daily AIDE report for $FQDN"
[email protected]
QUIETREPORTS=no
COMMAND=update
COPYNEWDB=yes
...

Now that AIDE is installed, and the basic preferences are in place, its now time to check out the main configuration files. The default configuration from the upstream provider should give you a reasonable default configuration. But what if you wanted to add your website documentroot to this so you can keep track of what files are changing on your website? The Debian/Ubuntu way of configuring AIDE is a bit different from the CentOS/RHEL method.

All the configuration files resides in /etc/aide/aide.conf.d/. The number of the file appears to be used by the AIDE wrapper to use for deciding which order to process these files. The AIDE documentation seems to indicate that the most general rules should be processed last, so I’ll default to creating my servers profile with 50_aide_CUSTOM-RULES.

So lets say I want to monitor my documentroot, here is how this would be setup:

[root@web01 ~]# vim /etc/aide/aide.conf.d/50_aide_CUSTOM-RULES
...
/var/www/vhosts/domain.com FULL
...

Now AIDE will be keeping track of our website. But adding your site may lead to very noisy reports because most websites implement caching. So this now becomes a balancing act to exclude directories that change often, yet retain enough of your sites critical content. We could just leave the entire directory in AIDE, but I know I personally don’t want to read a change report that contains 1,000 changes every day. So in the case of this wordpress site, I exclude the cache directory by appending the following to my custom configuration:

[root@web01 ~]# vim /etc/aide/aide.conf.d/50_aide_CUSTOM-RULES
...
/var/www/vhosts/domain.com Full
!/var/www/vhosts/domain.com/web/wp-content/cache
...

The “!” means NOT to monitor that specific directory. You will need to run AIDE a few times and fine tune the configuration before you get a report that is useful for your specific needs.

Anytime a change is made to your AIDE configuration, you need to rebuild the AIDE run time configuration, and initialize the database. You do that by:

[root@web01 ~]# update-aide.conf
[root@web01 ~]# aideinit -y -f

Now, try making a basic change to /etc/hosts, then run a check on AIDE to see if it detects the change and emails out the report:

[root@web01 ~]# /etc/cron.daily/aide

If you wanted to just quickly test AIDE to ensure it picks up your changes, but won’t commit them to baseline, you can perform a one-time scan by:

[root@web01 ~]# aide.wrapper

To receive nightly AIDE reports, no further configuration is needed since Ubuntu/Debian already setup a cron job that will run AIDE automatically in /etc/cron.daily/aide. This will run whenever your system normally runs the cron.daily jobs, which is defined in /etc/crontab.

Posted below is an example report that AIDE would send me via email daily:

This is an automated report generated by the Advanced Intrusion Detection 
Environment on web01.domain.com started at 2016-03-07 13:16:35.

AIDE returned with exit code 7. Added, removed and changed files detected!
AIDE post run information
output database /var/lib/aide/aide.db.new was copied to /var/lib/aide/aide.db as requested by cron job configuration
End of AIDE post run information

AIDE produced no errors.

Output of the daily AIDE run (83 lines):
AIDE 0.15.1 found differences between database and filesystem!!
Start timestamp: 2016-03-07 13:16:35

Summary:
  Total number of files:	77937
  Added files:			2
  Removed files:		3
  Changed files:		7


---------------------------------------------------
Added files:
---------------------------------------------------

f++++++++++++++++: /var/log/aide/aide.log.0
d++++++++++++++++: /var/www/vhosts/domain.com/new

---------------------------------------------------
Removed files:
---------------------------------------------------

f----------------: /var/www/vhosts/domain.com/blah
f----------------: /var/www/vhosts/domain.com/test
d----------------: /var/www/vhosts/domain.com/test1

---------------------------------------------------
Changed files:
---------------------------------------------------

f   p.g    . A. .: /var/log/aide/aide.log
d =.... mc.. .. .: /var/spool/postfix/active
d =.... mc.. .. .: /var/spool/postfix/incoming
d =.... mc.. .. .: /var/spool/postfix/maildrop
F =.... mc.. ..  : /var/spool/postfix/public/pickup
F =.... mc.. ..  : /var/spool/postfix/public/qmgr
d =.... mc.. .. .: /var/www/vhosts/domain.com

---------------------------------------------------
Detailed information about changes:
---------------------------------------------------


File: /var/log/aide/aide.log
 Perm     : -rw-------                       , -rw-r-----
 Gid      : 0                                , 4
 ACL      : old = A:
----
user::rw-
group::---
other::---
----
                  D: 
            new = A:
----
user::rw-
group::r--
other::---
----
                  D: 

Directory: /var/spool/postfix/active
 Mtime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23
 Ctime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23

Directory: /var/spool/postfix/incoming
 Mtime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23
 Ctime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23

Directory: /var/spool/postfix/maildrop
 Mtime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23
 Ctime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23

FIFO: /var/spool/postfix/public/pickup
 Mtime    : 2016-03-07 13:12:37              , 2016-03-07 13:17:37
 Ctime    : 2016-03-07 13:12:37              , 2016-03-07 13:17:37

FIFO: /var/spool/postfix/public/qmgr
 Mtime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:36
 Ctime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:36

Directory: /var/www/vhosts/domain.com
 Mtime    : 2016-03-07 13:03:25              , 2016-03-07 13:16:17
 Ctime    : 2016-03-07 13:03:25              , 2016-03-07 13:16:17

End of AIDE output.

The check was done against /var/lib/aide/aide.db with the following characteristics:
 Size     : 13041865
 Bcount   : 25480
 Mtime    : 2016-03-07 13:13:23
 Ctime    : 2016-03-07 13:13:23
 Inode    : 273628
 RMD160   : bIthG3Q5FiJmj4CIYdASjJx5Ygc=
 TIGER    : omto0nb3/oIqIiKHEjnbhjvXeGdfycbV
 SHA256   : VJPGKy61GxGfcSrjJFbrP879y/skJaiQ
 SHA512   : 7pz3FdYh8TvoNOqjxWBToZQNG6oxmrrp
 CRC32    : 1dYwqA==
 HAVAL    : LBFzyApqoYn7ogzoROG5FpneBO1s7R3p
 GOST     : iJ1tWPLtYaxxoFDHZEW8gxCS3/pVlS1G

The AIDE run created a new database /var/lib/aide/aide.db.new with the following characteristics:
 Size     : 13041834
 Bcount   : 25480
 Inode    : 273627
 RMD160   : 4TKRFSc0nt/VGDVvPEY8U6YNzaw=
 TIGER    : o4RzDHHWBlH+Zt3P7vI8GHHgGV1OecrC
 SHA256   : Gher/aINaU8r73/lQEWLQQSsKqP7sGjO
 SHA512   : D0/w3S6NOLZHw7D7dt1QxYBXe6miP5hF
 CRC32    : 5SRdpg==
 HAVAL    : pe7+ai57TPpW34NjJgTQxs+cQsFJ9zq0
 GOST     : RrIiyspbpKEb5wEGSG2HTYM7N6NUtKSv

End of AIDE daily cron job at 2016-03-07 13:18, run time 102 seconds

So this reports tells me that a log file for AIDE was rotated out, a new folder was created in my DocumentRoot called new, and the files/folders blah, test, and test1 where removed from my DocumentRoot.

Please remember that utilizing a tool to provide file integrity monitoring is only one part of a defense in depth strategy. There is no silver bullet for system security, but every layer you add will increase your security footprint which helps you with taking a proactive approach to security.

Using AIDE for file integrity monitoring (FIM) on CentOS

PCI-DSS 3.1 section 10.5.5 has the following requirement:

Use file-integrity monitoring or change-detection software on logs to ensure that existing log data cannot be changed without generating alerts (although new data being added should not cause an alert).

For large solutions, I would suggest using a well known tool such as Tripwire Enterprise. However many small to mid size companies that have a small footprint within their card holder data environment (CDE), may not be able to afford this. So what can companies use to meet this requirement? Implement AIDE (Advanced Intrusion Detection Environment).

Taken from the projects website, AIDE creates a database from the regular expression rules that it finds from the config file(s). Once this database is initialized it can be used to verify the integrity of the files.

AIDE is a very simple (yet powerful) program that runs from cron checking your files (typically once a night), and it will scan your system looking for any changes in the directories its monitoring. There are a number of different ways to use this program, but I’ll outline one that I like to use.

My requirements:
1. I want the reports to run nightly.
2. All change reports are emailed to me so I can archive them for a year offsite.
3. Have the database automatically commit the additions, deletions, and changes to baseline each time its ran.

In the event my system was compromised, I want to ensure that the malicious user was not able to modify, or delete my previous reports. Therefore, I choose not to store them on the machine. Its true that once the malicious user gained access to my system, they could change my AIDE config on me, but at least my previous reports will be intact which should help me when determining what malicious changes this user made to my server. Please note that I am making an assumption here that you are already backing up your system nightly, which would include your AIDE database! If you do not currently have a backup strategy in place, get one. Tools such as AIDE helps identify what files a malicious user may have changed, but if they completely crippled the system, you will need to restore from backups.

Setting up AIDE is fairly straight forward. It exists in most of package repositories out there including most variants of Linux and BSD.

On Linux based systems, you can install it by:

[root@web01 ~]# yum install aide

Once you have AIDE installed, the default configuration from the upstream provider should give you a reasonable default aide.conf. But what if you wanted to add your website documentroot to this so you can keep track of what files are changing on your website? Well, we simple add the directory to the aide.conf by including:

[root@web01 ~]# vim /etc/aide.conf
...
/var/www/vhosts/domain.com
...

Now AIDE will be keeping track of our website. But adding your site may lead to very noisy reports because most websites implement caching. So this now becomes a balancing act to exclude directories that change often, yet retain enough of your sites critical content. We could just leave the entire directory in AIDE, but I know I personally don’t want to read a change report that contains 1,000 changes every day. So in the case of this wordpress site, I exclude the cache directory by appending the following to my custom configuration:

[root@web01 ~]# vim /etc/aide.conf
...
/var/www/vhosts/domain.com
!/var/www/vhosts/domain.com/web/wp-content/cache
...

The “!” means NOT to monitor that specific directory. You will need to run AIDE a few times and fine tune the configuration before you get a report that is useful for your specific needs.

On CentOS, I had to change the following setting in /etc/aide.conf for the initialization to work:

[root@web01 ~]# vim /etc/aide.conf
...
# Whether to gzip the output to database
gzip_dbout=no
...

Once you have your configuration tuned for your specific purposes, you first my initialize the database to create a baseline before you can start getting change reports. You do this by:

[root@web01 ~]# aide --init
[root@web01 ~]# mv -f /var/lib/aide/aide.db.new /var/lib/aide/aide.db

Now, try making a basic change to /etc/hosts, then run a check on AIDE to see if it detects the change:

[root@web01 ~]# aide --check

If you are like me and would prefer not to have to log into 10 servers a day to run and view the reports, you can configure cron to run the report, and email you the results daily, while committing the changes to baseline. If you choose to go this route, it is critical that you review your change reports as they come in because we are essentially committing every change to the baseline. Here is how I configure cron:

[root@web01 ~]# crontab -e
# Perform daily change report
0 3 * * * /usr/sbin/aide --update | mail -s "AIDE Audit Report : web01.example.com" [email protected]

# Initialize the AIDE database once a day:
30 3 * * *  nice -19 /usr/sbin/aide --init;mv -f /var/lib/aide/aide.db.new /var/lib/aide/aide.db

Posted below is an example report that AIDE would send me via email daily:

AIDE found differences between database and filesystem!!
Start timestamp: 2012-09-13 01:24:05

Summary:
Total number of files: 57620
Added files: 1
Removed files: 1
Changed files: 1

---------------------------------------------------
Added files:
---------------------------------------------------

added: /var/spool/cron/root

---------------------------------------------------
Removed files:
---------------------------------------------------

removed: /etc/.aide.conf.swp

---------------------------------------------------
Changed files:
---------------------------------------------------

changed: /etc/aide.conf

--------------------------------------------------
Detailed information about changes:
---------------------------------------------------

File: /etc/aide.conf
Size : 2381 , 2390
Mtime : 2012-09-13 01:24:05 , 2012-09-13 01:24:05
Ctime : 2012-09-13 01:24:05 , 2012-09-13 01:24:05
MD5 : b+qbBDYEPesd+NCR1VRQHQ== , rG5pNPghdweedpU/c0ieHw==
RMD160 : T081ixhqik4efC3dfeCOBDCKpP4= ,
qe8MV0eteklAKmlZ5LTubaOUNKo=
SHA256 : g4jstEtfU8BNu+43jkrxJc9Cpr2SABZj ,
a65iaV54XR4vu8/zbA4Tdfe2U+W5uPNY

So this reports tells me that root’s crontab was added, a swap file for aide.conf was removed, and I updated the /etc/aide.conf recently.

Please remember that utilizing a tool to provide file integrity monitoring is only one part of a defense in depth strategy. There is no silver bullet for system security, but every layer you add will increase your security footprint which helps you with taking a proactive approach to security.