When shipping Nginx Access logs to Logstash, I recommend formatting the log lines in JSON straight out of Nginx. This speeds up filtering in Logstash later. It does come with two caveats though;

  1. Nginx escapes many characters that may appear in your URLs such as slashes, tabs and double-quotes. There is a way to deal with them which I may discuss in another post.
  2. The Access logs on the client server will be larger due to the additional JSON characters (keys, double-quotes, commas, braces and brackets).

If you can live with those two caveats then let’s log some JSON.

Here is the default Combined Access Log definition as it ships with Nginx:

The output would look like:


To write this same line as JSON here’s the custom definition I’ll call json:

And the output would look like:

 

That’s a lot more characters (80 vs. 188 characters) and if you have a high-traffic site, those extra bytes will require more disk storage. The benefit in reduced processing time by Logstash can easily justify the extra overhead. Check-out my post on Automating Logstash Configurations for handling the JSON format. Cheers!