Rackspace Cloud Monitoring syntax examples

This guide is just a quick reference article displaying the alarm criteria needed for certain checks. Most of these are available in the Rackspace control panel, and some are customized taken from the contrib github repo.

CPU

This check monitors the CPU for high CPU usage. The example below will send a warning message when the usage is at or over 90%, and will send a warning when the CPU usage is at or over 95%

Label:

:set consecutiveCount=5

if (metric['usage_average'] > 95) {
  return new AlarmStatus(CRITICAL, 'CPU usage is #{usage_average}%');
}

if (metric['usage_average'] > 90) {
  return new AlarmStatus(WARNING, 'CPU usage is #{usage_average}%');
}

return new AlarmStatus(OK, 'CPU usage is #{usage_average}%');

Memory

This check monitors your memory and swap usage. If your system has less than 5% memory available, and less than 10% of swap available, it will throw an alarm.

Label:

if (metric['swap_total'] > 0 && percentage(metric['swap_used'], metric['swap_total']) > 90
    && percentage(metric['actual_used'], metric['total']) > 95) {
  return new AlarmStatus(CRITICAL, 'Less than 5% of memory and 10% of swap available');
}

if (metric['swap_total'] == 0 && percentage(metric['actual_used'], metric['total']) > 95) {
  return new AlarmStatus(CRITICAL, 'Less than 5% of memory available');
}

return new AlarmStatus(OK, 'More than 5% of memory available');

Load Average

This checks the load average on the server. If the 15 minute load average is greater than 20, it will create an alarm.

Label: High Load Average

if (metric['15m'] > 20) {
  return new AlarmStatus(CRITICAL, '15 Minute Load Average is #{15m}');
}

return new AlarmStatus(OK, '15 Minute Load Average is #{15m}');

Filesystem

This one is broken down into 2 alarms. One for checking for avaiable space, and the other checking to see if the filesystem is in read only mode.

Please keep in mind that each alarm below should be a separate alarm!

Label: Low Filesystem Space

if (percentage(metric['used'], metric['total']) > 90) { 
    return new AlarmStatus(CRITICAL, 'Less than 10% free space available.'); 
    } 
if (percentage(metric['used'], metric['total']) > 80) { 
    return new AlarmStatus(WARNING, 'Less than 20% free space available.'); 
    } 
return new AlarmStatus(OK, 'Greater than 80% free space available.');

Label: Check for read only filesystem

if (metric['options'] regex ".*ro.*") {
return new AlarmStatus(CRITICAL, "Read-Only Filesystem");
}
return new AlarmStatus(OK, 'Filesystem in Read-Write mode.');

MySQL Replication Check

This is a custom agent plugin, so you need to download the plugin on your server first:

mkdir -p /usr/lib/rackspace-monitoring-agent/plugins
cd /usr/lib/rackspace-monitoring-agent/plugins/
wget https://raw.github.com/racker/rackspace-monitoring-agent-plugins-contrib/master/mysql_replication.py
chmod 755 mysql_replication.py

Now register the plugin with Cloud Monitoring:

curl -i -X POST -H 'Host: monitoring.api.rackspacecloud.com' -H 'Accept-Encoding: gzip,deflate' -H 'X-Auth-Token: YOUR_API_TOKEN' -H 'Content-Type: application/json; charset=UTF-8' -H 'Accept: application/json' --data-binary '{"label": "MySQL Replication Check", "type": "agent.plugin", "details": {"args": ["arg1"],"file": "mysql_replication.py"}}'  --compress 'https://monitoring.api.rackspacecloud.com:443/v1.0/YOUR_ACCOUNT_NUMBER/entities/ENTITY_ID/checks'

Finally, apply the alert criteria.

Label: MySQL Replication Check

if (metric['SLAVE_STATUS'] != 'ONLINE') {
  return new AlarmStatus(CRITICAL, 'MySQL Replication is OFFLINE.');
}

if (metric['SLAVE_STATUS'] == 'ONLINE' && metric['SECONDS_BEHIND_MASTER'] >= 120 && metric['SECONDS_BEHIND_MASTER'] < 300) {
  return new AlarmStatus(WARNING, 'MySQL Replication ONLINE but Slave is more than 2 minutes behind Master.');
}

if (metric['SLAVE_STATUS'] == 'ONLINE' && metric['SECONDS_BEHIND_MASTER'] >= 300) {
  return new AlarmStatus(CRITICAL, 'MySQL Replication ONLINE but Slave is more than 5 minutes behind Master.');
}

return new AlarmStatus(OK, 'MySQL Replication is ONLINE');

Holland Check

If you have Holland installed, you can monitor your nightly MySQL dumps to ensure no errors have been returned. It also checks to ensure MySQL is running, and that a valid /root/.my.cnf exists:

This is a custom agent plugin, so you need to download the plugin on your server first:

mkdir -p /usr/lib/rackspace-monitoring-agent/plugins
cd /usr/lib/rackspace-monitoring-agent/plugins/
wget https://raw.github.com/racker/rackspace-monitoring-agent-plugins-contrib/master/holland_mysqldump.py
chmod 755 holland_mysqldump.py

Now register the plugin with Cloud Monitoring:

raxmon-checks-create --entity-id=YOUR_ENTITY  --label=Holland --type=agent.plugin --username=YOUR_USERNAME --api-key=YOUR_API_KEY --details=file=holland_mysqldump.py

Finally, apply the alert criteria. Please keep in mind that each alarm below should be a separate alarm!

Label: Holland Log

if (metric['sql_ping_succeeds'] == 'true' && 
    metric['sql_creds_exist'] == 'true' && 
    metric['sql_status_succeeds'] == 'true' && 
    metric['dump_age'] < 172800 && 
    metric['error_count'] > 0) { 
  return new AlarmStatus(CRITICAL, 'holland-plugin: #{last_error}.'); 
} 
return new AlarmStatus(OK, 'holland-plugin: No errors found in most recent log entries.');

Label: MySQL Authenticates

if (metric['sql_ping_succeeds'] == 'true' && 
    metric['sql_creds_exist'] == 'true' && 
    metric['sql_status_succeeds'] == 'false') { 
  return new AlarmStatus(CRITICAL, 'holland-plugin: MySQL credentials do not authenticate.'); 
} 
return new AlarmStatus(OK, 'holland-plugin: MySQL credentials authenticate.');

Label: MySQL Credentials Exist

if (metric['sql_ping_succeeds'] == 'true' && 
    metric['sql_creds_exist'] == 'false') { 
  return new AlarmStatus(CRITICAL, 'holland-plugin: MySQL credentials file does not exist.'); 
} 
return new AlarmStatus(OK, 'holland-plugin: MySQL credentials file exists.');

Label: MySQL Running

if (metric['sql_ping_succeeds'] == 'false') { 
  return new AlarmStatus(CRITICAL, 'holland-plugin: MySQL is not running.'); 
} 
return new AlarmStatus(OK, 'holland-plugin: MySQL is running');

Label: Recent Backup Exists

if (metric['sql_ping_succeeds'] == 'true' && 
    metric['sql_creds_exist'] == 'true' && 
    metric['sql_status_succeeds'] == 'true' && 
    metric['dump_age'] > 172800) { 
  return new AlarmStatus(CRITICAL, 'holland-plugin: mysqldump file is older than 2d.'); 
} 
return new AlarmStatus(OK, 'holland-plugin: mysqldump file age is less than 2d.');

Process Check

This is a quick and dirty plugin I wrote for when you have to be notified if something is not running on the server, such as Lsyncd or Memcached since both these can’t (Lsyncd) or shouldn’t (Memcached) be listening on the public interface. Any process in the process list can be monitored with this.

This is a custom agent plugin, so you need to download the plugin on your server first:

mkdir -p /usr/lib/rackspace-monitoring-agent/plugins
cd /usr/lib/rackspace-monitoring-agent/plugins
wget https://raw.github.com/racker/rackspace-monitoring-agent-plugins-contrib/master/process_mon.sh
chmod 755 process_mon.sh

Now register the plugin with Cloud Monitoring:

curl -i -X POST -H 'Host: monitoring.api.rackspacecloud.com' -H 'Accept-Encoding: gzip,deflate' -H 'X-Auth-Token: YOUR_API_TOKEN' -H 'Content-Type: application/json; charset=UTF-8' -H 'Accept: application/json' --data-binary '{"label": "Process Check", "type": "agent.plugin", "details": {"args": ["PROCESS_NAME"],"file": "process_mon.sh"}}'  --compress 'https://monitoring.api.rackspacecloud.com:443/v1.0/YOUR_ACCOUNT/entities/YOUR_ENTITY/checks'

Finally, apply the alert criteria in the Rackspace control panel:

if (metric['process_mon'] == 0) {
return new AlarmStatus(CRITICAL, 'Process not running.');
}

return new AlarmStatus(OK, 'Process running normally.');

Rackspace Cloud API – Enable SSL Termination on existing cloud load balancer

The purpose of this post is to show how you can enable SSL termination on an existing cloud load balancer through the API. Using the API will allow you to script deployments so you can avoid having to use the control panel. This also provides you the building blocks for understanding deployment automation.

This guide will show you how to enable SSL termination on an existing cloud load balancer as described in my previous post: Rackspace Cloud API – Create cloud load balancers

Feel free to review http://docs.rackspace.com for learning about all the possible operations that can be done through the API.

In this example, we are going to enable SSL termination on an existing cloud load balancer called lb.example.com. The domain we are load balancing is www.example.com. Please note that enabling SSL termination on a Rackspace cloud load balancer costs more then a regular cloud load balancer!

SPECIAL NOTE: I advise against using SSL termination if you are passing any PII (personally identifiable information) or other sensitive data through the cloud load balancer to the Cloud server. The transmission will only be encrypted from the clients browser to the load balancer. From there, the cloud load balancer will send the request in clear text through the rackspace network to your cloud server.

When working with the API, I like to use a tool called httpie to simplify things a bit. You can install this by:

yum install httpie

Now that we have httpie installed, lets get an auth token from the API:

echo '{"auth": {"RAX-KSKEY:apiKeyCredentials": {"username": "YOUR_USERNAME","apiKey":"YOUR_API_KEY"}}}' | http post https://identity.api.rackspacecloud.com/v2.0/tokens

The token you need will be listed next to “id” field as shown below

        "token": {
            "expires": "2013-07-09T23:17:08.634-05:00", 
            "id": "2334aasdf5555j3hfhd22245dhsr", 
            "tenant": {
                "id": "123456", 
                "name": "123456"

To simplify things moving forward, we will set some local variables that we’ll use when communicating with the API:

export token="YOUR_API_TOKEN_RECEIVED_ABOVE"
export account="YOUR_RACKSPACE_CLOUD_ACCOUNT_NUMBER"
export lb_endpoint="https://ord.loadbalancers.api.rackspacecloud.com/v1.0"

NOTE: Change the endpoints region accordingly (ord or dfw).

For the purposes of this guide, we will be creating a self signed SSL certificate. In production environments, you will want to purchase a SSL certificate through a CA.

We will create our self signed SSL certificate by using the openssl command:

openssl req -x509 -nodes -newkey rsa:2048 -keyout example.com.tmp.key -out example.com.cert -days 1825
Generating a 2048 bit RSA private key
........+++
.....................................+++
writing new private key to 'example.com.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:California
Locality Name (eg, city) []: Los Angeles
Organization Name (eg, company) [Internet Widgits Pty Ltd]:My Company LLC
Organizational Unit Name (eg, section) []:IT
Common Name (e.g. server FQDN or YOUR name) []:*.example.com
Email Address []:

Now we must covert the key so the API will be able to use it:

openssl rsa -in example.com.tmp.key -out example.com.key
rm -f example.com.tmp.key

Format the key and cert so it is more friendly for the API. Be sure to save the output as we will need it for the next step:

while read line; do echo -n "$line\n"; done < example.com.key
while read line; do echo -n "$line\n"; done < example.com.cert

Now, lets setup the json file that contains our certificate information:

cat << EOF > example.com.cert.json
{
   "certificate":"-----BEGIN CERTIFICATE-----\nMIIblahblahblahblah\n-----END CERTIFICATE-----",
   "enabled":true,
   "secureTrafficOnly":false,
   "privatekey":"-----BEGIN RSA PRIVATE KEY-----\nMIICWblahblahblahblah\n-----END RSA PRIVATE KEY-----",
   "intermediateCertificate":"",
   "securePort":443}
EOF

Finally, lets execute the json file to enable SSL on your pre-existing cloud load balancer:

http put $lb_endpoint/$account/loadbalancers/YOUR_CLOUD_LOAD_BALANCER_ID/ssltermination X-Auth-Token:$token @example.com.cert.json

SSL termination on your existing load balancer has now been enabled.

Rackspace Cloud API – Create cloud load balancers

The purpose of this post is to show how you can build Rackspace cloud load balancers using the API. Building via the API will allow you to script deployments so you can avoid having to use the control panel. This also provides you the building blocks for understanding deployment automation.

This guide will only show you how to create a cloud load balancer. Feel free to review http://docs.rackspace.com for learning about all the possible operations that can be done through the API.

In this example, we are going to deploy a single cloud load balancer called lb.example.com that will be directing HTTP traffic to two web servers:

test01.example.com
test02.example.com

When working with the API, I like to use a tool called httpie to simplify things a bit. You can install this by:

yum install httpie

Now that we have httpie installed, lets get an auth token from the API:

echo '{"auth": {"RAX-KSKEY:apiKeyCredentials": {"username": "YOUR_USERNAME","apiKey":"YOUR_API_KEY"}}}' | http post https://identity.api.rackspacecloud.com/v2.0/tokens

The token you need will be listed next to “id” field as shown below

        "token": {
            "expires": "2013-07-09T23:17:08.634-05:00", 
            "id": "2334aasdf5555j3hfhd22245dhsr", 
            "tenant": {
                "id": "123456", 
                "name": "123456"

To simplify things moving forward, we will set some local variables that we’ll use when communicating with the API:

export token="YOUR_API_TOKEN_RECEIVED_ABOVE"
export account="YOUR_RACKSPACE_CLOUD_ACCOUNT_NUMBER"
export lb_endpoint="https://ord.loadbalancers.api.rackspacecloud.com/v1.0"

NOTE: Change the endpoints region accordingly (ord or dfw).

Lets prep a json file that we’ll be using to build the cloud load balancer:

cat << EOF > lb.example.com.json
{
    "loadBalancer": {
        "name": "lb.example.com",
        "port": 80,
        "protocol": "HTTP",
        "virtualIps": [
            {
                "type": "PUBLIC"
            }
         ],
        "nodes": [
            {
                "address": "10.123.123.121",
                "port": 80,
                "condition": "ENABLED"
            }
        ]
    }
}
EOF

Now we execute the build by:

http post $lb_endpoint/$account/loadbalancers X-Auth-Token:$token @lb.example.com.json

This will return the VIP and id of the new load balancer as shown below:

                "address": "123.123.123.123", 
        "id": 123456, 
        "name": "lb.example.com", 

For the purposes of this guide, I left out adding the second server initially. So here is how we can add it to the load balancer so it will route traffic between test01.example.com and test02.example.com. First create a json file that contains test02.example.com:

cat << EOF > nodes.json
{"nodes": [
{
"address": "10.123.123.123",
"port": 80,
"condition": "ENABLED",
"type":"PRIMARY"
}
]
}
EOF

Now add this node to the load balancer:

http post $lb_endpoint/$account/loadbalancers/149275/nodes X-Auth-Token:$token @nodes.json

And your done! You can verify everything looks correct by running:

http get $lb_endpoint/$account/loadbalancers/YOUR_LOAD_BALANCER_ID X-Auth-Token:$token

Rackspace Cloud API – Create NextGen cloud servers

The purpose of this post is to show how you can build Rackspace Next Generation cloud servers using the API. Building via the API will allow you to script server builds so you can avoid having to use the control panel. This also provides you the building blocks for understanding deployment automation.

This guide will only show you how to create a cloud server. Feel free to review http://docs.rackspace.com for learning about all the possible operations that can be done through the API.

In this example, we are going to build 2 512M CentOS 6.4 Cloud Servers via the Rackspace Cloud API. The servers will be named:

test01.example.com
test02.example.com

When working with the API, I like to use a tool called httpie to simplify things a bit. You can install this by:

yum install httpie

Now that we have httpie installed, lets get an auth token from the API:

echo '{"auth": {"RAX-KSKEY:apiKeyCredentials": {"username": "YOUR_USERNAME","apiKey":"YOUR_API_KEY"}}}' | http post https://identity.api.rackspacecloud.com/v2.0/tokens

The token you need will be listed next to “id” field as shown below

        "token": {
            "expires": "2013-07-09T23:17:08.634-05:00", 
            "id": "2334aasdf5555j3hfhd22245dhsr", 
            "tenant": {
                "id": "123456", 
                "name": "123456"

To simplify things moving forward, we will set some local variables that we’ll use when communicating with the API:

export token="YOUR_API_TOKEN_RECEIVED_ABOVE"
export account="YOUR_RACKSPACE_CLOUD_ACCOUNT_NUMBER"
export endpoint="https://ord.servers.api.rackspacecloud.com/v2/"

NOTE: Change the endpoints region accordingly (ord or dfw).

Now, lets see what images are available. I’m looking for a CentOS 6.4 image:

http get $endpoint/$account/images/detail X-Auth-Token:$token

The id of the CentOS 6.4 image in this case is:

            "id": "e0ed4adb-3a00-433e-a0ac-a51f1bc1ea3d", 

We wanted a 512M server, so we must find the flavors id:

http get $endpoint/$account/flavors X-Auth-Token:$token

This shows that the 512M flavor has the id of:

            "id": "2", 

All the information has been collected. Time to prep 2 json files that we’ll be using to build the 2 servers:

cat << EOF > test01.example.com.json
{
    "server" : {
        "name" : "test01.example.com",
        "imageRef" : "e0ed4adb-3a00-433e-a0ac-a51f1bc1ea3d",
        "flavorRef" : "2"
    }
}
EOF

cat << EOF > test02.example.com.json
{
    "server" : {
        "name" : "test02.example.com",
        "imageRef" : "e0ed4adb-3a00-433e-a0ac-a51f1bc1ea3d",
        "flavorRef" : "2"
    }
}
EOF

Finally, we have everything we need to begin the builds. Execute the build by:

http post $endpoint/$account/servers @test01.example.com.json X-Auth-Token:$token
http post $endpoint/$account/servers @test02.example.com.json X-Auth-Token:$token

When you run each POST statement above, 2 fields will be returned by the API. Be sure to record these somewhere:
– adminPass : This is your servers root password
– id : This is your servers id number that will be referenced next.

A new server is not useful without knowing its IP address. After a few minutes pass, you can retrieve the IP address by running:

http get $endpoint/$account/servers/YOUR_SERVER_ID X-Auth-Token:$token
http get $endpoint/$account/servers/YOUR_OTHER_SERVER_ID X-Auth-Token:$token

The 2 relevant fields you will need for this example are posted below:

        "accessIPv4": "123.123.123.123", 
                    "addr": "10.123.123.123", 

Now you can SSH into your server using the IP and admin password that have been returned by the API.