Taken directly from Munin’s website:
Munin is a networked resource monitoring tool that can help analyze resource trends and "what just happened to kill our performance?" problems. It is designed to be very plug and play. A default installation provides a lot of graphs with almost no work.
As the paragraph above suggests, Munin is a really good tool for being able to spot anomalies in the servers performance that may help point a system administrator in the right direction for determining where the bottlenecks are coming into play.
Munin can be installed on each node as a stand alone instance, or you can setup a Munin master server, then simply have the clients (Munin nodes) report back to the Munin master.
If would would prefer to have a stand alone instance of Munin on each node, then follow the instructions for setting up the Munin Master server. This guide is currently making the assumption that you already have Apache setup and working on your servers.
Munin Master Setup
To install the latest stable version of Munin, run the following:
# CentOS 6 / RedHat 6 [root@munin-master ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm [root@munin-master ~]# yum install munin # CentOS 7 / RedHat 7 [root@munin-master ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm [root@munin-master ~]# yum install munin
To configure the Munin master, first uncomment the following lines:
[root@munin-master ~]# vim /etc/munin/munin.conf ... dbdir /var/lib/munin htmldir /var/www/html/munin logdir /var/log/munin rundir /var/run/munin tmpldir /etc/munin/templates ...
Now setup the permissions so Munin can access the directory:
[root@munin-master ~]# mkdir -p /var/www/html/munin [root@munin-master ~]# chown munin:munin /var/www/html/munin
To protect our performance metrics from being seen by the public, protect the web directory by:
[root@munin-master ~]# yum install httpd-tools [root@munin-master ~]# htpasswd -c /etc/munin/munin-htpasswd serverinfo New password: Re-type new password: Adding password for user serverinfo
Then uncomment, add, or modify the following fields:
# Apache only
[root@munin-master ~]# vim /etc/httpd/conf.d/munin.conf
...
Order allow,deny
Allow from all
Options None
...
AuthUserFile /etc/munin/munin-htpasswd
AuthName "Munin"
AuthType Basic
require valid-user
...
[root@munin-master ~]# service httpd restart
# Nginx only
[root@munin-master ~]# vim /etc/nginx/conf.d/default.conf
server {
...
location /munin/static/ {
alias /etc/munin/static/;
expires modified +1w;
}
location /munin {
auth_basic "Restricted";
auth_basic_user_file /etc/munin/munin-htpasswd;
alias /var/www/html/munin/;
expires modified +310s;
}
...
}
[root@munin-master ~]# service nginx restart
Finally, we need to update the name in the host tree to properly identify the Munin Master. So we will be changing the name from localhost.localdomain to MuninMaster. You could change to this to anything you like, such as a FQDN like web01.domain.com if you like.
[root@munin-master ~]# vim /etc/munin/munin.conf
...
[MuninMaster]
address 127.0.0.1
use_node_name yes
...
Restart Apache and Munin so the changes register:
[root@munin-master ~]# service httpd restart [root@munin-master ~]# service munin-node restart
As Munin only refreshes the stats once every 5 minutes via cron, manually run it the first time so you can check out Munin without having to wait:
[root@munin-master ~]# su - munin -s /bin/bash -c 'test -x /usr/bin/munin-cron && /usr/bin/munin-cron'
Then view your Munin stats by navigating your browser to:
http://serverip/munin
How to add a node to the Munin Master
Here is where we can install additional servers to the Munin Master, so you can view all the graphs from one interface.
First, install the munin-node package on your additional server:
# CentOS 6 / RedHat 6 [root@web01 ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm [root@web01 ~]# yum install munin-node # CentOS 7 / RedHat 7 [root@web01 ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm [root@web01 ~]# yum install munin-node
Now configure munin-node to accept inbound connections from your Munin Master server. About halfway down in the file, below the allow ^127\.0\.0\.1$, add the IP address of your Munin Master server. For example, if the IP address of the Munin Master is 192.168.1.100, then the entry would be:
[root@web01 ~]# vim /etc/munin/munin-node.conf ... allow ^127\.0\.0\.1$ allow ^192\.168\.1\.100$ ...
Then restart Munin:
[root@web01 ~]# service munin-node restart
Allow the Munin Master server to connect through the firewall:
[root@web01 ~]# vim /etc/sysconfig/iptables ... -A INPUT -s 192.168.1.100/24 -p tcp --dport 4949 -m comment --comment "Allow Munin Master server access" -j ACCEPT ... [root@web01 ~]# service iptables restart
Back on the Munin Master server, we need to tell Munin to query the new Munin Node for information. You can change the FQDN to anything you like, and also be sure to update the IP address to use the IP of your Munin Node:
[root@munin-master ~]# vim /etc/munin/munin.conf
...
[web01.domain.com]
address 192.168.1.200
use_node_name yes
...
Then restart Munin:
[root@munin-master ~]# service munin-node restart
Finally, view your Munin stats by navigating your browser to back to your Munin Master server.
Enable additional plugins
Munin comes preinstalled with a number of plugins, but many are disabled since every server runs different things.
To see what plugins can be used on your server, run:
[root@web01 ~]# munin-node-configure --suggest Plugin | Used | Suggestions ------ | ---- | ----------- acpi | no | no [cannot read /proc/acpi/thermal_zone/*/temperature] amavis | no | no apache_accesses | yes | yes apache_processes | no | no apache_volume | yes | yes ...
To break this down:
- If 'Used' column says 'yes', then that means the plugin is already in use - If 'Used' column says 'no' and 'Suggestions' says 'no', then the plugin cannot be used. - If 'Used' column says 'no' and 'Suggestions' says 'yes', then you can enable that plugin
Enabling plugins is pretty easy. Lets say you want to enable the iostat plugin:
[root@web01 ~]# ln -s /usr/share/munin/plugins/iostat /etc/munin/plugins
Then restart Munin:
[root@web01 ~]# service munin-node restart
Enable additional plugins – Apache
Munin has the ability to graph Apache accesses, processes, and overall volume for Apache. In order for this to work, mod_status needs to be enabled with ExtendedStatus set to on, and needs to be accessible from the server at: http://localhost/server-status?auto. An example server-status config is shown below:
# RHEL / CentOS
[root@web01 ~]# vim /etc/httpd/conf.d/status.conf
<IfModule mod_status.c>
#
# ExtendedStatus controls whether Apache will generate "full" status
# information (ExtendedStatus On) or just basic information (ExtendedStatus
# Off) when the "server-status" handler is called. The default is Off.
#
ExtendedStatus On
# Allow server status reports generated by mod_status,
# with the URL of http://servername/server-status
# Uncomment and change the ".example.com" to allow
# access from other hosts.
#
<Location /server-status>
SetHandler server-status
Order deny,allow
Deny from all
Allow from localhost ip6-localhost 127.0.0.1
<IfModule mod_rewrite.c>
RewriteEngine off
</IfModule>
# AuthUserFile /etc/httpd/status-htpasswd
# AuthName "Password protected"
# AuthType Basic
# Require valid-user
# Allow password-less access for allowed IPs
Satisfy any
</Location>
</IfModule>
# RHEL / CentOS
[root@web01 ~]# service httpd restart
Now test to ensure the server-status module is working:
[root@web01 ~]# curl localhost/server-status ... cache type: SHMCB, shared memory: 512000 bytes, current sessions: 0
subcaches: 32, indexes per subcache: 133
index usage: 0%, cache usage: 0%
total sessions stored since starting: 0
total sessions expired since starting: 0
total (pre-expiry) sessions scrolled out of the cache: 0
total retrieves since starting: 0 hit, 0 miss
total removes since starting: 0 hit, 0 miss
</td></tr> </table> </body></html>
Then confirm that the Munin plugin works by running the plugin manually. Confirm the output says ‘yes’ as shown below:
[root@web01 ~]# /usr/share/munin/plugins/apache_accesses autoconf yes [root@web01 ~]# /usr/share/munin/plugins/apache_processes autoconf yes [root@web01 ~]# /usr/share/munin/plugins/apache_volume autoconf yes
Now enable the Apache plugins for Munin and restart the service:
[root@web01 ~]# ln -s /usr/share/munin/plugins/apache_accesses /etc/munin/plugins/ [root@web01 ~]# ln -s /usr/share/munin/plugins/apache_processes /etc/munin/plugins/ [root@web01 ~]# ln -s /usr/share/munin/plugins/apache_volume /etc/munin/plugins/ [root@web01 ~]# service munin-node restart
Finally, wait about 5-10 minutes for the stats to be generated then check the Munin graphs to see if they are now registering the new data. Otherwise you can test the plugins manually to confirm they are returning stats by:
[root@web01 ~]# /etc/munin/plugins/apache_accesses accesses80.value 1300 [root@web01 ~]# /etc/munin/plugins/apache_processes busy80.value 1 idle80.value 8 free80.value 58 [root@web01 ~]# /etc/munin/plugins/apache_volume volume80.value 5093376
Enable additional plugins – Nginx
Munin has the ability to graph Nginx requests and nginx status for Nginx. In order for this to work, the Nginx stub_status module needs to be enabled so the following page will work: http://localhost/nginx_status. An example nginx_status configuration is shown below:
[root@web01 ~]# vim /etc/nginx/conf.d/nginx_status.conf
server {
listen 80 default_server;
access_log off;
server_name _;
server_name_in_redirect off;
root /var/www/html;
location /nginx_status {
# Enable Nginx stats
stub_status on;
# Disable logging for stats
access_log off;
# Security: Only allow access from authorized IP's
allow 127.0.0.1;
# allow xx.xx.xx.xx;
# Deny everyone else
deny all;
}
}
[root@web01 ~]# service nginx restart
Now test to ensure the stub_status module is working with curl:
[root@web01 ~]# curl localhost/nginx_status Active connections: 1 server accepts handled requests 1 1 1 Reading: 0 Writing: 1 Waiting: 0
Then confirm the Munin plugin works by running the plugin manually. Confirm the output says ‘yes’ as shown below:
[root@web01 ~]# /usr/share/munin/plugins/nginx_request autoconf yes [root@web01 ~]# /usr/share/munin/plugins/nginx_status autoconf yes
Enable the Nginx plugins for Munin and restart the service:
[root@web01 ~]# ln -s /usr/share/munin/plugins/nginx_request /etc/munin/plugins/ [root@web01 ~]# ln -s /usr/share/munin/plugins/nginx_status /etc/munin/plugins/ [root@web01 ~]# service munin-node restart
Finally, wait about 5-10 minutes for the stats to be generated then check the Munin graphs to see if the graphs are now registering the new data. Otherwise you can test the plugins manually to confirm they are returning stats by:
[root@web01 ~]# /etc/munin/plugins/nginx_request request.value 3 [root@web01 ~]# /etc/munin/plugins/nginx_status total.value 1 reading.value 0 writing.value 1 waiting.value 0
Enable additional plugins – MySQL
Munin has the ability to graph a number of metrics for MySQL, including commands, caches, buffers, innodb stats, table locks, tmp tables, etc. In order for the plugin to work, Munin will need to be able to access MySQL, and there are also a few dependencies that need to be taken care of. First, install the needed perl modules on the Munin node:
# RHEL / CentOS 6 and 7 [root@db01 ~]# yum install perl-Class-DBI-mysql perl-DBD-mysql libdbi-dbd-mysql
Setup the MySQL user that Munin will use to log into MySQL to pull the stats:
[root@db01 ~]# mysql mysql> GRANT PROCESS,SUPER on *.* to 'munin'@'127.0.0.1' IDENTIFIED BY 'YOUR_SECURE_PASSWORD_HERE'; mysql> GRANT PROCESS,SUPER on *.* to 'munin'@'localhost' IDENTIFIED BY 'YOUR_SECURE_PASSWORD_HERE'; mysql> flush privileges; mysql> quit
Setup some configurations within Munin to let it know how to access MySQL:
[root@db01 ~]# vim /etc/munin/plugin-conf.d/munin-node ... [mysql*] env.mysqladmin /usr/bin/mysqladmin env.mysqluser munin env.mysqlpassword YOUR_SECURE_PASSWORD_HERE ... [root@db01 ~]# vim /etc/munin/plugin-conf.d/mysql_ [mysql_*] env.mysqlconnection DBI:mysql:information_schema;host=127.0.0.1;port=3306 env.mysqluser munin env.mysqlpassword YOUR_SECURE_PASSWORD_HERE env.cachenamespace munin_mysql_pri [root@db01 ~]# vim /etc/munin/plugin-conf.d/mysql_innodb [mysql_innodb] env.warning 0 env.critical 0
Now enable the Munin MySQL plugins:
[root@db01 ~]# for i in `/usr/share/munin/plugins/mysql_ suggest`; do ln -s /usr/share/munin/plugins/mysql_ /etc/munin/plugins/mysql_$i ; done
Restart Munin so the new settings go into effect:
[root@db01 ~]# service munin-node restart
Finally, wait about 5-10 minutes for the stats to be generated then check the Munin graphs to see if the graphs are now registering the new data.
Enable additional plugins – Memcached
Munin has the ability to graph commands, current values and network traffic for Memcached. There are also a few dependencies that need to be taken care of. First, install the needed perl modules on the Munin node:
# RHEL / CentOS 6 and 7 [root@memcache01 ~]# yum install perl-Cache-Memcached
Then setup some configurations within Munin to let it know how to access Memcached:
[root@memcache01 ~]# vim /etc/munin/plugin-conf.d/munin-node ... [memcached_*] env.host 127.0.0.1 env.port 11211 ...
Now enable the Munin Memcached plugins:
[root@memcache01 ~]# ln -s /usr/share/munin/plugins/memcached_ /etc/munin/plugins/memcached_bytes [root@memcache01 ~]# ln -s /usr/share/munin/plugins/memcached_ /etc/munin/plugins/memcached_counters [root@memcache01 ~]# ln -s /usr/share/munin/plugins/memcached_ /etc/munin/plugins/memcached_rates
Restart Munin so the new setting take effect:
[root@memcache01 ~]# service munin-node restart
Finally, wait about 5-10 minutes for the stats to be generated then check the Munin graphs to see if the graphs are now registering the new data.
Troubleshooting – Connectivity issues
If Munin is not graphing data, it could be because Munin cannot connect to the master server. Some basic things to check would be to ensure Munin is running on the master server and node server. Ensure that port 4949 is opened inbound to the Munin master from the node. Finally on the Munin master, confirm that /etc/munin/munin-node.conf has an allow block for the Munin node server.
You can test connectivity by running the following, substituting localhost out accordingly if your connecting from a Munin node server:
[root@web01 ~]# telnet localhost 4949 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. # munin node at localhost.localdomain
Troubleshooting – See what Munin plugins are enabled
To see which plugins are enabled on your Munin node, you can quickly test for it by issuing the ‘list’ command via telnet:
[root@web01 ~]# telnet localhost 4949 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. # munin node at localhost.localdomain list apache_accesses apache_processes apache_volume cpu df df_inode entropy forks fw_conntrack fw_forwarded_local fw_packets if_err_eth0 if_eth0 interrupts irqstats load memcached_bytes memcached_counters memcached_rates memory ...
Troubleshooting – Testing a plugin manually
If a graph is showing up blank, its useful to test the plugin directly to ensure it is able to grab the data. Using the ‘df’ plugin as the example, it can be ran manually by:
[root@web01 ~]# munin-run df _dev_mapper_VolGroup_lv_root.value 16.9176920622854 _dev_sda1.value 27.6994797122402
Troubleshooting – Plugin not generating graph data
If a plugin is not generating stats within Munin, try running the plugin manually to see if it will return stats on the command line. Equally important, ensure that the application is getting some traffic for the plugin to graph. Finally, keep in mind that it can take up to 15 minutes for a plugin to start graphing, assuming that you see stats when running it manually.