Log Everything

Log entries provide insight in the behavior of your app. By logging you are making it (more or less) possible to rewind time and replay what happened in your app at a certain moment in time.

Log entries help you debug issues, detect “anomalies” (a fancy word for “stuff that should not happen”), and they provide a way to track changes over time.

There are at least two ways to use log entries:

• Line by line: each line tells you something specific.
• Over a period of time: to spot a trend.

Each log entry can have a different purpose. To name two:

• At what moment a certain event happened.
• At what moment a certain value changed into another value.

You should not be afraid to log “too much”.

Within a typical web app you should log:

• Server-side and client-side exceptions.
• Failures in connecting to 3rd party interfaces.
• Unhandled fall-through cases in your code.
• Successful and unsuccessful sign attempts.
• Changes to important entities such as account or billing properties.

Should I use “structured logging”?

If you can, yes but it’s hard to enforce a single structure across your company.

The least you can do is to create some kind of a “dictionary” that your development team should use to apply structure to its log entries.

Examples:

You might decide to post JSON to your log:

{ "datestamp": 1551452743, "severity":"error", "system": "messaging", "subsystem": "sms-provider", "error-code": 1001, "error-msg": "Insufficient balance" }

In this case your “dictionary” contains the words datestamp, severity, system, subsystem, error-code, and error-msg.

If you can’t use JSON, simply use a hashtag-like notation:

1551452743,"#severity:error","#system:messaging","#subsystem:sms-provider","#error-code:1001","#error-msg:Insufficient balance"

Make sure you can easily parse the different parts of your log entry (this is why the various parts are enclosed by double quotes and separated by commas).

How do I write log entries?

Simply by writing to stdout and by letting your favorite logging framework take care of the rest. The beauty of using stdout is that you can inspect the log on your local development machine without needing any tools other than cat or tail.

Your logging framework should take care of storing, processing, rotating, and archiving logs. If you ever decide to switch to a different logging framework then you don’t need to touch your code.

Having said that, a simple wrapper around stdout is handy, enabling you to write entries with different severities with ease:

Logger.info "This is informational"

Logger.error "This is not good"

In case you embrace structured logging:

Logger.error { "system": "messaging", "subsystem": "sms-provider", "error-code": 1001, "error-msg": "Insufficient balance" }

Note that the severity is derived from the .error method name and the datestamp is based on the time of calling.

What about monitoring & alerting?

Monitoring & alerting rely on logs to carry out their work. Start by setting up proper logging. Once this is in place, set up one or more monitoring tools which ingest your logs. Instruct your monitoring tool to look for patterns, which in turn triggers alerts.

Recommended reading

• On monitoring & alerting
• https://12factor.net/logs ("Treat logs as event streams")
• https://www.loggly.com/blog/30-best-practices-logging-scale/