JSON logging using Logback
Logs are useful. Logs that provide us with insights are more useful :), and if they do this easily, they’re even more so.
Most logging implementations assume that humans will be consuming log statements. So they’re essentially formatted string (think printf) so that humans can easily read them. In these situations, what we’re really doing is creating too much log data for humans to consume to get any particularly useful insights. People then try to solve this overload problem by being cautious about what they log, the idea being that lesser information will be easier for humans to handle. Unfortunately, we can’t know beforehand what information we may need to debug an issue. What always ends up happening is that some important piece of information is missed out.
Remember how we can program machines to consume lots of data and provide better insights? So instead of creating log files for humans to consume, we need to create them for machines.
One log format that works very well is JSON. JSON objects aren’t particularly human readable, but machines love them. We can use any JSON library to parse our logs.. So lets see how we can use Logback to log in JSON format in a clojure project. Best part is we do not have to change the way we log, we just have to configure Logback to print our logs in JSON format.
First we need to add these dependencies to our project.clj
Also add these global exclusions to project.clj
Unfortunately, we need to add these exclusions because of the way maven handles dependencies. You might be able to avoid these exclusions, if your project does not depend on any third party libraries. If your project is dependent on third party libraries it is better to add these exclusions, so you know exactly which version of these logging related libs are in your classpath.
Create logback.xml in your classpath.
Logback uses this config file to do these following things,
- Add your logs to logs/development.log file
- Use JSON formatter
- Change Log times to be in UTC
- Add new line after every log statement
- Roll the log file based on Time and size of the log file.
Now your logs should be JSON format. This is an example log taken from my project.
You can ship these logs to log management services like Loggly, or write your own program and get better insights.
Have fun logging.