/ New Features

Fly Middleware: Unified Logging with Syslog, HTTP Log, Loggly and Papertrail

Fresh for you this fine Monday are four new Middleware. Together, these features bring unified logging for all of your applications running behind Fly. Let's dig in.

First, we'll look at the common philosophy behind the four features. Logging is a key pillar of Developer Operations. Whereas monitoring is a point-in-time look at the status of your environment - how much RAM you're using, how much CPU is being utilized - logging is a running historical record of events. There are two popular logging methods that we'll discuss in brief.

The first is the Syslog. If it's a networked device, it likely has a syslog - a method of sending system events to a syslog server, the store of system events. Your edge servers, application servers, and routers all record a Syslog. Log messages contain basic information: a timestamp, an IP address, the message itself. There isn't an agreed upon standard for Syslogs -- it's a method of getting data from a source to a collection.

The second are HTTP logs. They contain the same information that would be contained within a Syslog, except they travel over HTTP instead of TCP or UDP, like Syslog would. With HTTP comes additional data, contained within HTTP headers.

The key difference is that Syslog packets are simple strings of text that travel over their own protocol. The packes are lightweight, made up of: PRIORITY, HEADER, and MSG. An HTTP Log can be up to 5MB in size, while a Syslog packet must come in under 1KB -- which is quite ample, for a log message. For security, HTTP logs can be sent over SSL, while Syslogs can be sent with TLS via TCP.

With Fly, logging data for each of your site requests can be forwarded wherever you'd like, over either Syslog or HTTP. For Syslog, activate the Syslog Middleware, then configure it like so:

syslog-middleware

You can select your protocol - TCP, TCP+TLS, or UDP - and your choice of output: Common Log or JSON. To route your request logs to an HTTP endpoint, configure the HTTP Log Middleware like so:

http-log-middleware

You can choose your method: PUT, POST, or PATCH. Similar to Syslog, you can output in Common Log or JSON format. Once you've enabled either Middleware, your requests will show up in your logging repository for your perusing pleasure -- neat stuff!

A Loggly Papertrail

Brilliant products have been created to help you organize, parse, and maintain your ever-growing collection of logging data. You never know how far you'll need to go back or what you'll need to find, so when the time comes to dive into your logs it's helpful to have complimentary tools. Two of our favourites are Loggly and Papertrail. To make it easier to apply them, we've created Loggly HTTP Log and Papertrail Syslog Middlware.

Connecting either one is simple: Loggly supports HTTP logs. Login to Loggly, click Source Setup, then search for HTTP. You'll see two entries: HTTP/S Bulk Endpoint and HTTP/S Event Endpoint -- choose whichever best fits your use-case. If you don't know, default to bulk!

For Papertrail, take your URL and Port, then simply plug the values into Fly. The TLS certificates will be taken care of for you. Like the Syslog and HTTP Log Middleware, the moment you click enable the Logs Runneth Over.

Certified Unified

Fly is an Application Delivery Network. Through it, you attach a hostname, then stack your various backends and services on top. You could, for example, have Heroku, Amazon S3, and a set of powerful AWS Lambda functions attached to onehostname.com. Each application that's running through Fly will sit atop your hostname on a subfolder: /app/ or /app/compress, in the case of one of your serverless functions.

With the Middleware enabled, any request directed at your domain will be logged. That means that each source backend or service will have its request logs accumulated into one clean pipe, which can be directed wherever you'd like for storage and analysis. If, like our example case, you're hosting an application in conjunction with serverless functions, static storage, or other functional services, you don't need to be concerned about routing the output of each of those -- just one pipe.

Fly is free to sign up, as are your first 2 million requests and 100GB of transfer every month. To conclude, a lovely quote from the introduction to Syslogs' RFC: 3164:

Since the beginning, life has relied upon the transmission of messages. For the self-aware organic unit, these messages can relay many different things. The messages may signal danger, the presence of food or the other necessities of life, and many other things. In many cases, these messages are informative to other units and require no acknowledgement. As people interacted and created processes, this same principle was applied to societal communications. As an example, severe weather warnings may be delivered through any number of channels - a siren blowing, warnings delivered over television and radio stations, and even through the use of flags on ships. - C. Lonvick, RFC: 3164

Kellen Evan Person

@thegoodroot

Kellen Evan Person

A polite, forest-dwelling Canadian who enjoys coding and writing. He's spent near two decades building web applications and strives to keep development fun and light-hearted.

North Vancouver, Canada