Ship Logs from Docker with the Logz.io Fluentd Proxy

The past year has been significant for continued development of both DevOps practices and new developments across the open source community. To that end, Logz.io is moving forward with renewed support for the Fluentd log shipper. This new proxy will serve as an alternative to Filebeat and Logstash, which recently moved away from open source licensing. Additionally, this integration utilizes an HTTP proxy instead of the SOCKS5 proxy necessary for Filebeat.

This tutorial will show you how to configure your environment to ship logs from Docker containers to Logz.io, with Fluentd sitting in its own separate container. We’ll go over pulling the Docker image of – and configuring a containerized version of – Fluentd.

Once deployed, every container in your environment (including the Fluentd container) will write logs to a common file. The data will go directly – or through a proxy server – over HTTP or HTTPS to Logz.io.

Filebeat has some important limitations, starting with its use of the SOCKS5 proxy, which is less popular than HTTP. That shortcoming limits which tools you can use alongside Filebeat. Additionally, using a TCP connection for SOCKS can represent a security risk for users.

Compatibility is also an issue. It was already difficult to use Filebeat with anything other than the Elastic Stack. But starting with Version 7.13, Filebeat is incompatible with past versions of the Elastic Stack itself. This presents a major burden on developers, who now have to constantly keep up with Elastic Stack updates to ensure their own self-hosted versions of the stacks continue to work. While Filebeat 7.13 was still under an open-source Apache 2 license, this restrictive approach effectively shutters Filebeat to other past open-source releases of the ELK Stack.

Most importantly, Fluentd remains open source. Retaining the ability to use and contribute to the software freely, or to download it and make what you will of it, embodies the spirit of open source. OSS projects seek to provide the most indispensable tools to the world at large and make them as compatible as possible with other similarly-licensed projects. 

Installing and Configuring the Fluentd Container

First, pull the Docker image for the Logz.io Fluentd proxy:

docker pull logzio/fluentd-docker-logs

Next, start the Fluentd container with the below code. Please note that Fluentd’s default port is 8071. Also, other data will need your manual input. 

Start the Fluentd container with the below command, replacing <<LISTENER-HOST>> and <<LOG-SHIPPING-TOKEN>> with the correct info (available in your Logz.io account):

  docker run -it --rm \
   --name fluentd-docker-logs \
   -v $(pwd)/log:/fluentd/log \
   -v /var/lib/docker/containers:/var/lib/docker/containers:/var/lib/docker/containers:ro \
   -v /var/run/docker.sock:/var/run/docker.sock:ro \
   -p 5001:5001 \
   -e LOGZIO_LOG_LISTENER="https://<<LISTENER-HOST>>:8071" \
   -e LOGZIO_LOG_SHIPPING_TOKEN=<<LOG-SHIPPING-TOKEN>> \
   -e LOGZIO_TYPE=docker-fluentd \
   -e LOGZIO_PROXY_URI=nil \
   -e LOGZIO_PROXY_CERT=nil \
   logzio/fluentd-docker-logs

If you are using a proxy server in the middle of the shipping route, replace nil in the following two lines with the correct info:

   -e LOGZIO_PROXY_URI=nil \
   -e LOGZIO_PROXY_CERT=nil \

From there, you should be able to filter your logs by docker-fluentd in the Logz.io app.

Advanced Fluentd Configuration

You might want to get more specific and customize your log output. We have several other tags you can add to the start command with the -e parameter. Specify the value after the = sign on each parameter. 

Here’s the list of those optional settings, with their default values used as filler:

   -e LOGZIO_INCLUDE_REGEX=.+ \
   -e LOGZIO_SLOW_FLUSH_LOG_THRESHOLD=20.0 \
   -e LOGZIO_BUFFER_TYPE=file \
   -e LOGZIO_BUFFER_PATH=/var/log/Fluentd-buffers/stackdriver.buffer \
   -e LOGZIO_OVERFLOW_ACTION=block \
   -e LOGZIO_CHUNK_LIMIT_SIZE=2M
   -e LOGZIO_QUEUE_LIMIT_LENGTH=6 \
   -e LOGZIO_FLUSH_INTERVAL=5s \
   -e LOGZIO_RETRY_MAX_INTERVAL=30s \
   -e LOGZIO_FLUSH_THREAD_COUNT=2 \
   -e LOGZIO_LOG_LEVEL=info \

Keeping Fluent with the Latest

To keep up with the latest releases, integrations and news from Logz.io, subscribe to our blog. Expect more news in the next few weeks related to other popular open source tools.