Monitor Phusion Passenger with AWS CloudWatch

• infra
Back

If you aren't already familiar, Phusion Passenger is a mature, performant, and secure web application server. It has broad support for a number of application platforms, from Ruby (Rack) to Python, Node.js, Meteor, and more. I have evaluated and used other application servers in varying workloads, but Passenger has consistently been the most reliable, performant, and feature rich offering available.

In this article, I aim to demonstrate how to pair Phusion Passenger's built in administration tools with AWS CloudWatch to monitor your application server's active instance pool over time, as well as the critical "request queue", which anyone running Phusion Passenger in production should keep an eye on (more on that in a future blog post).

Phusion Passenger offers a rich administration API, accessed via HTTP requests made to the locally running instance process listening on a dynamically generated port. These HTTP requests require authentication, which is handled using HTTP basic auth, with credentials provided by the passenger Ruby gem. I'm using v5.1.8 in this example.

In short, we:

  1. Load the Passenger Ruby libraries, including the AdminTools module.
  2. Use the AdminTools::InstanceRegistry to get an Array of all locally running Passenger instances.
  3. For our selected instance, call the Agent Core API /pools.txt endpoint, to return the formatted instance status data.
  4. Use regex to extract our important metrics.
  5. Report those metrics to AWS CloudWatch, using our Instance ID (gathered via the EC2 metadata service) as our key dimension.

This Ruby script is a working demonstration of this process, which will work on a EC2 instance with the aws-sdk and passenger gems installed, however you manage your dependencies (probably Bundler). This same logic can be implemented in any language that supports these libraries, however.

#!/usr/bin/env ruby
#
# Report Instance Count and Request Queue Size metrics from locally running Phusion Passenger instance.
# Author: Andrew Page <andrew@andrewpage.me>
# ==
# Phusion Passenger: https://www.phusionpassenger.com

require 'net/http'
require 'phusion_passenger'
require 'aws-sdk'

PhusionPassenger.locate_directories
PhusionPassenger.require_passenger_lib 'admin_tools/instance_registry'
include PhusionPassenger::AdminTools

# Extract a parameter from the Passenger pool status output.
def extract_metric(parameter, output)
    pattern = %r{#{parameter}\s+:\s+(?<result>\d+)}
    match   = output.match(pattern)
    match['result'] if match
end

# Get the Passenger pool status for our active instance,
def get_pool_status(instance)
    request = Net::HTTP::Get.new('/pool.txt')
 
    # Authenticate the HTTP request to the Passenger API.
    request.basic_auth('ro_admin', instance.read_only_admin_password)
 
    # Request pool status
    response = instance.http_request('agents.s/core_api', request)
    response.body
end

# Generate the metric to report to CloudWatch.
def build_cloudwatch_metric(metric_name, value, instance_id:, timestamp: Time.now)
    {
        metric_name:        metric_name,
        timestamp:          timestamp,
        value:              value,
        storage_resolution: 1,
        dimensions:         [
            {
                name:   'InstanceId',
                value:  instance_id
            }
        ]
    }
end

instances = InstanceRegistry.new.list
if instances.count > 1
    $stderr.puts 'Multiple Passenger instances... not sure which to use. Exiting.'
    exit(1)
end

# Get pool status from Passenger instance.
instance  = instances.first
output    = get_pool_status(instance)

# Extract key metrics from Passenger output.
timestamp           = Time.now
process_count       = extract_metric('Processes', output)
request_queue_size  = extract_metric('Requests in queue', output)

# Get current instance ID
instance_id = Net::HTTP.get(URI('http://169.254.169.254/latest/meta-data/instance-id'))

# Report metrics to AWS CloudWatch.
cloudwatch = Aws::CloudWatch::Client.new
cloudwatch.put_metric_data(
    namespace:    'Passenger',
    metric_data:  [
        build_cloudwatch_metric('ProcessCount',     process_count,      instance_id:  instance_id,  timestamp: timestamp),
        build_cloudwatch_metric('RequestQueueSize', request_queue_size, instance_id:  instance_id,  timestamp: timestamp)
    ]
)

$stderr.puts "Reported Passenger process count of #{process_count} and request queue size of #{request_queue_size} to CloudWatch."

This script should probably be invoked in some sort of scheduled manner (probably cron) to provide continuous reporting of metrics. However you execute this script, the executing user must have user permissions or group membership in line with those running the Passenger process itself, as global read permissions are denied for a number of important files.

Resources

© Andrew PageRSS