Kubernetes Logging with Fluentd and Logz.io

Kubernetes is developing so rapidly, that it has become challenging to stay up to date with the latest changes (Heapster has been deprecated!). The ecosystem around Kubernetes has exploded with new integrations developed by the community, and the field of logging and monitoring is one such example.

The topic of logging containers orchestrated by Kubernetes with the ELK Stack has already been written about extensively both on the Logz.io blog and elsewhere. The most common approach we’re seeing now, is hooking up Kubernetes with what is increasingly being referred to as the EFK Stack — Elasticsearch, Fluentd and Kibana. Deploying Fluentd as a daemonset, users can spin up a Fluentd pod for each node in their Kubernetes cluster with the correct configurations to forward data to their Elasticsearch deployment.

There are various daemonset configurations available but the one used here to send logs to Logz.io’s ELK Stack is based on a daemonset configuration provided by the kind folks at Treasure Data — the company driving the development of Fluentd and Fluent Bit. We’ve added some basic parsing capabilities on top of this configuration, such as support for multi-line processing for exception stack traces (java, js, csharp, python, go, ruby, php) and parsing the log field as message.

For those new to Kubernetes, steps 1-2 will help you with setting up the demo environment and are most likely superfluous for those with a running Kubernetes cluster. These steps describe setting up Minikube, kubectl and deploying a basic demo app for generating some simple log data. Putting in place the logging infrastructure is described in subsequent steps.

Step 1: Setting up your Kubernetes development environment

First, install kubectl, the CLI for running commands against Kubernetes clusters. In this case, I’m installing kubectl’s binary on my Mac using cURL:

curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s 

I’m then making the binary executable and moving to my PATH:

chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl

Second, install Minikube.

Minikube enables you to easily run Kubernetes locally as a single-node cluster inside a VM. Be sure to first install a hypervisor (I’m using VirtualBox).

For Mac, you can use this cURL command:

curl -Lo minikube 
-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

Start Minikube with:

minikube start

Finally, run the following kubectl command to make sure both kubectl and Minikube were installed correctly and that the former can connect to your Kubernetes cluster:

kubectl cluster-info

You should be seeing a URL response:

Kubernetes master is running at
KubeDNS is running at

Our last step is to deploy the Minikube dashboard — a UI that allows you to easily deploy your apps to your Kubernetes cluster, troubleshoot it, and manage the cluster itself along with all the  relevant resources.

minikube dashboard

The dashboard opens up in your browser automatically:


Step 2: Deploying a demo app using Minikube

Let’s start with deploying a basic demo application on our Kubernetes cluster. For this purpose, I’ll use Docker’s voting app — a basic app built of five services for handling online voting.

Clone the repo and use the provided specs file to deploy to your Kubernetes cluster:

git clone https://github.com/dockersamples/example-voting-app.git

Enter the voting app’s directory:

cd /example-voting-app

Next, create the vote namespace

kubectl create namespace vote
kubectl create -f k8s-specifications/

deployment.extensions "db" created
service "db" created
deployment.extensions "redis" created
service "redis" created
deployment.extensions "result" created
service "result" created
deployment.extensions "vote" created
service "vote" created
deployment.extensions "worker" created

After a few minutes, all services, deployments and pods should be up and running, and to access the voting app, simply open your browser using the cluster IP and port 31000:


You can review all your pods either using kubectl or the dashboard:

kubectl get pods

Or, if you receive an error that reads No resources found in default namespace, try this:

kubectl get pods --all-namespaces

Then your pods should appear like this:

NAME                      READY     STATUS    RESTARTS   AGE
db-86b99d968f-fkxqg       1/1       Running   0          15m
redis-659469b86b-gtkpx    1/1       Running   0          15m
result-59f4f867b8-6ntrg   1/1       Running   0          15m
vote-54f5f76b95-jmmwx     1/1       Running   0          15m
worker-56578c48f8-ljqph   1/1       Running   0          15m

Step 3: Creating a Fluentd Daemonset

As mentioned above, the method we’re going to use for hooking up our development cluster with Logz.io involves deploying a Fluentd as a daemonset. A close look at the YAML reveals that with a few tweaks to the environment variables, the same daemonset can be used to ship logs to your own ELK deployment as well.

Create a new daemonset configuration file:

sudo vim daemonset.yaml

Use this configuration, and be sure to enter your Logz.io account token in the environment variables section:

apiVersion: extensions/v1beta1
kind: DaemonSet
  name: fluentd-logzio
  namespace: kube-system
    k8s-app: fluentd-logzio
    version: v1
    kubernetes.io/cluster-service: "true"
        k8s-app: fluentd-logzio
        version: v1
        kubernetes.io/cluster-service: "true"
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      - name: fluentd
        image: logzio/logzio-k8s:1.0.0
          - name:  LOGZIO_TOKEN
            value: "your logz.io account token"
          - name:  LOGZIO_URL
            value: "your logz.io host url" ##example:https://listener.logz.io:8071  
            memory: 200Mi
            cpu: 100m
            memory: 200Mi
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
    readOnly: true
      terminationGracePeriodSeconds: 30
      - name: varlog
          path: /var/log
      - name: varlibdockercontainers
          path: /var/lib/docker/containers

Create the Daemonset with:

kubectl create -f daemonset.yaml

You can, of course, use the dashboard as well for the same purpose. In any case, after a minute or two you will see a new pod deployed:

kubectl get pods

NAME                      READY     STATUS              RESTARTS   AGE
db-86b99d968f-fkxqg       1/1       Running             0          22m
fluentd-d7fc2             1/1       Running             0          1m
redis-659469b86b-gtkpx    1/1       Running             0          22m
result-59f4f867b8-6ntrg   1/1       Running             0          22m
vote-54f5f76b95-jmmwx     1/1       Running             0          22m
worker-56578c48f8-ljqph   1/1       Running             0          22m

In Logz.io, you will begin to see log data being generated by your Kubernetes cluster:

data kubernetes

Step 4: Visualizing Kubernetes logs in Kibana

As mentioned above, the image used by this daemonset knows how to handle exceptions for a variety of applications, but Fluentd is extremely flexible and can be configured to break up your log messages in any way and fashion you like depending on the type of logs being collected.

Also worthy of note is that this Fluentd image adds useful Kubernetes metadata to the logs which can come in handy in larger environments consisting of multiple nodes and pods. Below are a few examples of how you can leverage this metadata to gain visibility into your Kubernetes cluster with Kibana visualizations.

Metric visualizations are simple and are great for displaying simple stats related to your setup. For example, you can use a unique count aggregation of the kubernetes.container_name field to see how many containers you’ve got running in each pod.


You can do the same of course for the number of nodes running in your cluster.

Using line charts and a combination of count aggregations together with time histograms, we can get a nice picture of the logging pipeline in our cluster:


Monitoring the stderr output for error messages reporting, I can create a basic line chart showing me duplicate votes in our voting app:

1 line

We can add all these into a dashboard to get a nice overview of our Kubernetes cluster:



The EFK stack (Elasticsearch, Fluentd and Kibana) is probably the most popular method for centrally logging Kubernetes deployments. In fact, many would consider it a de-facto standard. The combination of an easily deployable and versatile log aggregator, a high-performing data store and a rich visualization tool is a powerful solution.

Of course, the log data generated by a single-node cluster deployed with Minikube on Mac does not do justice to the full potential of the stack — the visualizations above are simple examples and you can slice and dice your Kubernetes logs in any way you want.

We are always looking to improve, so we’d love for you guys to try out this daemonset and give us your feedback.


Observability at scale, powered by open source


2022 Gartner® Magic Quadrant for Application Performance Monitoring and Observability
Forrester Observability Snapshot.

Organize Your Kubernetes Logs On One Unified SaaS Platform

Learn More