Consul and Consul-Template are two powerful tools for updating server files and executing a command after the file has been updated. This makes it a great and lightweight tool for managing configuration files for applications that need to reload or restart after a configuration change.

In this post I will describe how to setup a Logstash configuration file for Nginx. I describe writing Nginx Access logs in JSON here. Installing Consul, Consul-Template and Logstash is beyond the scope of this post.

Logstash

Logstash is a data processing pipeline. It has Inputs, Filters and Outputs. Popular use is for log processing where the Input is a log line from a server, the Filter formats the line (and much more) and the Output sends the formatted line to a receiver like Elasticsearch. Configuration is done via file(s) that is read by Logstash at runtime. The management of these files can be done manually, using configuration management like Saltstack or my favorite, using Consul and Consul-Template.

For this example I will assume logstash-forwarder on the remote server has been configured to tail the nginx access log and shipped the log line(s) to the logstash server.


Add Keys to Consul

Input (Consul Key: services/logging/etc/logstash/conf.d/inputs)

Value:

Filter (Consul Key: services/logging/etc/logstash/conf.d/filters)

Value:

Output (Consul Key: services/logging/etc/logstash/conf.d/outputs)

Value:


Create Consul-Template Files

/etc/consul-template.d/configs/nginx.hcl

/etc/consul-template.d/templates/nginx.ctmpl

That’s it. Just restart Consul-Template and the logstash configuration file will be created and logstash restarted. If there were no errors you’ll start processing Nginx log lines. Cheers!

Tagged on: