Rackspace Cloud Monitoring syntax examples

This guide is just a quick reference article displaying the alarm criteria needed for certain checks. Most of these are available in the Rackspace control panel, and some are customized taken from the contrib github repo.

CPU

This check monitors the CPU for high CPU usage. The example below will send a warning message when the usage is at or over 90%, and will send a warning when the CPU usage is at or over 95%

Label:

:set consecutiveCount=5

if (metric['usage_average'] > 95) {
  return new AlarmStatus(CRITICAL, 'CPU usage is #{usage_average}%');
}

if (metric['usage_average'] > 90) {
  return new AlarmStatus(WARNING, 'CPU usage is #{usage_average}%');
}

return new AlarmStatus(OK, 'CPU usage is #{usage_average}%');

Memory

This check monitors your memory and swap usage. If your system has less than 5% memory available, and less than 10% of swap available, it will throw an alarm.

Label:

if (metric['swap_total'] > 0 && percentage(metric['swap_used'], metric['swap_total']) > 90
    && percentage(metric['actual_used'], metric['total']) > 95) {
  return new AlarmStatus(CRITICAL, 'Less than 5% of memory and 10% of swap available');
}

if (metric['swap_total'] == 0 && percentage(metric['actual_used'], metric['total']) > 95) {
  return new AlarmStatus(CRITICAL, 'Less than 5% of memory available');
}

return new AlarmStatus(OK, 'More than 5% of memory available');

Load Average

This checks the load average on the server. If the 15 minute load average is greater than 20, it will create an alarm.

Label: High Load Average

if (metric['15m'] > 20) {
  return new AlarmStatus(CRITICAL, '15 Minute Load Average is #{15m}');
}

return new AlarmStatus(OK, '15 Minute Load Average is #{15m}');

Filesystem

This one is broken down into 2 alarms. One for checking for avaiable space, and the other checking to see if the filesystem is in read only mode.

Please keep in mind that each alarm below should be a separate alarm!

Label: Low Filesystem Space

if (percentage(metric['used'], metric['total']) > 90) { 
    return new AlarmStatus(CRITICAL, 'Less than 10% free space available.'); 
    } 
if (percentage(metric['used'], metric['total']) > 80) { 
    return new AlarmStatus(WARNING, 'Less than 20% free space available.'); 
    } 
return new AlarmStatus(OK, 'Greater than 80% free space available.');

Label: Check for read only filesystem

if (metric['options'] regex ".*ro.*") {
return new AlarmStatus(CRITICAL, "Read-Only Filesystem");
}
return new AlarmStatus(OK, 'Filesystem in Read-Write mode.');

MySQL Replication Check

This is a custom agent plugin, so you need to download the plugin on your server first:

mkdir -p /usr/lib/rackspace-monitoring-agent/plugins
cd /usr/lib/rackspace-monitoring-agent/plugins/
wget https://raw.github.com/racker/rackspace-monitoring-agent-plugins-contrib/master/mysql_replication.py
chmod 755 mysql_replication.py

Now register the plugin with Cloud Monitoring:

curl -i -X POST -H 'Host: monitoring.api.rackspacecloud.com' -H 'Accept-Encoding: gzip,deflate' -H 'X-Auth-Token: YOUR_API_TOKEN' -H 'Content-Type: application/json; charset=UTF-8' -H 'Accept: application/json' --data-binary '{"label": "MySQL Replication Check", "type": "agent.plugin", "details": {"args": ["arg1"],"file": "mysql_replication.py"}}'  --compress 'https://monitoring.api.rackspacecloud.com:443/v1.0/YOUR_ACCOUNT_NUMBER/entities/ENTITY_ID/checks'

Finally, apply the alert criteria.

Label: MySQL Replication Check

if (metric['SLAVE_STATUS'] != 'ONLINE') {
  return new AlarmStatus(CRITICAL, 'MySQL Replication is OFFLINE.');
}

if (metric['SLAVE_STATUS'] == 'ONLINE' && metric['SECONDS_BEHIND_MASTER'] >= 120 && metric['SECONDS_BEHIND_MASTER'] < 300) {
  return new AlarmStatus(WARNING, 'MySQL Replication ONLINE but Slave is more than 2 minutes behind Master.');
}

if (metric['SLAVE_STATUS'] == 'ONLINE' && metric['SECONDS_BEHIND_MASTER'] >= 300) {
  return new AlarmStatus(CRITICAL, 'MySQL Replication ONLINE but Slave is more than 5 minutes behind Master.');
}

return new AlarmStatus(OK, 'MySQL Replication is ONLINE');

Holland Check

If you have Holland installed, you can monitor your nightly MySQL dumps to ensure no errors have been returned. It also checks to ensure MySQL is running, and that a valid /root/.my.cnf exists:

This is a custom agent plugin, so you need to download the plugin on your server first:

mkdir -p /usr/lib/rackspace-monitoring-agent/plugins
cd /usr/lib/rackspace-monitoring-agent/plugins/
wget https://raw.github.com/racker/rackspace-monitoring-agent-plugins-contrib/master/holland_mysqldump.py
chmod 755 holland_mysqldump.py

Now register the plugin with Cloud Monitoring:

raxmon-checks-create --entity-id=YOUR_ENTITY  --label=Holland --type=agent.plugin --username=YOUR_USERNAME --api-key=YOUR_API_KEY --details=file=holland_mysqldump.py

Finally, apply the alert criteria. Please keep in mind that each alarm below should be a separate alarm!

Label: Holland Log

if (metric['sql_ping_succeeds'] == 'true' && 
    metric['sql_creds_exist'] == 'true' && 
    metric['sql_status_succeeds'] == 'true' && 
    metric['dump_age'] < 172800 && 
    metric['error_count'] > 0) { 
  return new AlarmStatus(CRITICAL, 'holland-plugin: #{last_error}.'); 
} 
return new AlarmStatus(OK, 'holland-plugin: No errors found in most recent log entries.');

Label: MySQL Authenticates

if (metric['sql_ping_succeeds'] == 'true' && 
    metric['sql_creds_exist'] == 'true' && 
    metric['sql_status_succeeds'] == 'false') { 
  return new AlarmStatus(CRITICAL, 'holland-plugin: MySQL credentials do not authenticate.'); 
} 
return new AlarmStatus(OK, 'holland-plugin: MySQL credentials authenticate.');

Label: MySQL Credentials Exist

if (metric['sql_ping_succeeds'] == 'true' && 
    metric['sql_creds_exist'] == 'false') { 
  return new AlarmStatus(CRITICAL, 'holland-plugin: MySQL credentials file does not exist.'); 
} 
return new AlarmStatus(OK, 'holland-plugin: MySQL credentials file exists.');

Label: MySQL Running

if (metric['sql_ping_succeeds'] == 'false') { 
  return new AlarmStatus(CRITICAL, 'holland-plugin: MySQL is not running.'); 
} 
return new AlarmStatus(OK, 'holland-plugin: MySQL is running');

Label: Recent Backup Exists

if (metric['sql_ping_succeeds'] == 'true' && 
    metric['sql_creds_exist'] == 'true' && 
    metric['sql_status_succeeds'] == 'true' && 
    metric['dump_age'] > 172800) { 
  return new AlarmStatus(CRITICAL, 'holland-plugin: mysqldump file is older than 2d.'); 
} 
return new AlarmStatus(OK, 'holland-plugin: mysqldump file age is less than 2d.');

Process Check

This is a quick and dirty plugin I wrote for when you have to be notified if something is not running on the server, such as Lsyncd or Memcached since both these can’t (Lsyncd) or shouldn’t (Memcached) be listening on the public interface. Any process in the process list can be monitored with this.

This is a custom agent plugin, so you need to download the plugin on your server first:

mkdir -p /usr/lib/rackspace-monitoring-agent/plugins
cd /usr/lib/rackspace-monitoring-agent/plugins
wget https://raw.github.com/racker/rackspace-monitoring-agent-plugins-contrib/master/process_mon.sh
chmod 755 process_mon.sh

Now register the plugin with Cloud Monitoring:

curl -i -X POST -H 'Host: monitoring.api.rackspacecloud.com' -H 'Accept-Encoding: gzip,deflate' -H 'X-Auth-Token: YOUR_API_TOKEN' -H 'Content-Type: application/json; charset=UTF-8' -H 'Accept: application/json' --data-binary '{"label": "Process Check", "type": "agent.plugin", "details": {"args": ["PROCESS_NAME"],"file": "process_mon.sh"}}'  --compress 'https://monitoring.api.rackspacecloud.com:443/v1.0/YOUR_ACCOUNT/entities/YOUR_ENTITY/checks'

Finally, apply the alert criteria in the Rackspace control panel:

if (metric['process_mon'] == 0) {
return new AlarmStatus(CRITICAL, 'Process not running.');
}

return new AlarmStatus(OK, 'Process running normally.');