Tech behind Tech

Raw information. No finesse :)

JSON logging using Logback

with 2 comments


Logs are useful. Logs that provide us with insights are more useful :), and if they do this easily, they’re even more so.

Most logging implementations assume that humans will be consuming log statements. So they’re essentially formatted string (think printf) so that humans can easily read them. In these situations,  what we’re really doing is creating too much log data for humans to consume to get any particularly useful insights. People then try to solve this overload problem by being cautious about what they log, the idea being that lesser information will be easier for humans to handle. Unfortunately, we can’t know beforehand what information we may need to debug an issue. What always ends up happening is that some important piece of information is missed out.

Remember how we can program machines to consume lots of data and provide better insights? So instead of creating log files for humans to consume, we need to create them for machines.

One log format that works very well is JSON.  JSON objects aren’t particularly human readable, but machines love them. We can use any JSON library to parse our logs.. So lets see how we can use Logback to log in JSON format in a clojure project. Best part is we do not have to change the way we log, we just have to configure Logback to print our logs in JSON format.

First we need to add these dependencies to our project.clj

Also add these global exclusions to project.clj

Unfortunately, we need to add these exclusions because of the way maven handles dependencies. You might be able to avoid these exclusions, if your project does not depend on any third party libraries. If your project is dependent on third party libraries it is better to add these exclusions, so you know exactly which version of these logging related libs are in your classpath.

Create logback.xml in your classpath.

Logback uses this config file to do these following things,

  1. Add your logs to logs/development.log file
  2. Use JSON formatter
  3. Change Log times to be in UTC
  4. Add new line after every log statement
  5. Roll the log file based on Time and size of the log file.

Now your logs should be JSON format. This is an example log taken from my project.

You can ship these logs to log management services like Loggly, or write your own program and get better insights.

Have fun logging.

Written by Siva Jagadeesan

September 23, 2013 at 6:00 am

Posted in Clojure

Tagged with , , ,

2 Responses

Subscribe to comments with RSS.

  1. I would seriously advise the use of logstash-logback-encoder or logstash-log4j-encoder if you’re on log4j which outputs in a format that is directly ingerable by logstash, the de-facto standard for consuming JSON logs

    Pierre-Yves Ritschard

    September 23, 2013 at 8:03 am


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 146 other followers

%d bloggers like this: