Hosted Kubernetes services such as AKS were introduced to help engineers deal with the complexity involved in deploying and managing Kubernetes clusters. They do not cover, however, the task of monitoring Kubernetes and the services running on it. 

While some of the hosted Kubernetes services offer logging solutions, they do not offer all the functionality, flexibility and user-experience expected from a modern log management solution. The ELK Stack, or the EFK Stack to be more precise fills in these gaps by providing a Kubernetes-native logging experience — fluentd (running as a daemonset) to aggregate the different logs from the cluster, Elasticsearch to store the data and Kibana to slice and dice it.

This article will show how to hook up an AKS Kubernetes cluster with Logz.io’s ELK Stack. You’ll need two things for following the steps outlined below — an existing Kubernetes cluster on AKS and a Logz.io account. 

Step 1: Deploy a demo application on AKS

To simulate some load and generate log data, we’ll deploy a demo application on our AKS cluster. For this purpose, we’ll use the Azure voting app which includes two Kubernetes Services – a Redis instance and an external service for accessing the app. 

If you already have services running on AKS, feel free to skip directly to the next step.

So first, we’ll create a new manifest file containing the specs of deploying the app:

vim azure-vote.yaml

We’ll use this YAML file for deploying the app:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: azure-vote-back
spec:
 replicas: 1
 selector:
   matchLabels:
     app: azure-vote-back
 template:
   metadata:
     labels:
       app: azure-vote-back
   spec:
     nodeSelector:
       "beta.kubernetes.io/os": linux
     containers:
     - name: azure-vote-back
       image: redis
       resources:
         requests:
           cpu: 100m
           memory: 128Mi
         limits:
           cpu: 250m
           memory: 256Mi
       ports:
       - containerPort: 6379
         name: redis
---
apiVersion: v1
kind: Service
metadata:
 name: azure-vote-back
spec:
 ports:
 - port: 6379
 selector:
   app: azure-vote-back
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: azure-vote-front
spec:
 replicas: 1
 selector:
   matchLabels:
     app: azure-vote-front
 template:
   metadata:
     labels:
       app: azure-vote-front
   spec:
     nodeSelector:
       "beta.kubernetes.io/os": linux
     containers:
     - name: azure-vote-front
       image: microsoft/azure-vote-front:v1
       resources:
         requests:
           cpu: 100m
           memory: 128Mi
         limits:
           cpu: 250m
           memory: 256Mi
       ports:
       - containerPort: 80
       env:
       - name: REDIS
         value: "azure-vote-back"
---
apiVersion: v1
kind: Service
metadata:
 name: azure-vote-front
spec:
 type: LoadBalancer
 ports:
 - port: 80
 selector:
   app: azure-vote-front

To deploy the app, we’ll use:

kubectl apply -f azure-vote.yaml

We’ll get the following output:

deployment "azure-vote-back" created
service "azure-vote-back" created
deployment "azure-vote-front" created
service "azure-vote-front" created

To verify the app is working, we’re going to access it in our browser.  As part of the deployment process, a Kubernetes service exposes the app’s front end to the internet. This might take a minute or two, and we can follow the status of the deployment with:

kubectl get service azure-vote-front --watch

NAME               TYPE CLUSTER-IP             EXTERNAL-IP PORT(S) AGE
azure-vote-front   LoadBalancer 10.0.232.187   52.188.177.27 80:32515/TCP 1m

As soon as an external IP is available for the service, we’ll simply paste it in our browser:

As part of the deployment process, a Kubernetes service exposes the Azure app’s front end to the internet.

Step 2: Ship Kubernetes logs to Logz.io

Our cluster and the app deployed on it is now generating a mix of log data, all useful for gaining insight into how our environment is performing. Our next step is to ship this log data into Logz.io. 
To do this, we’ll use a daemonset that runs a fluentd pod on each node in our Kubernetes cluster.
Our first step is to store Logz.io credentials as a Kubernetes secret — our Logz.io account’s shipping token and Logz.io’s listener host.  We can find the shipping token in the Logz.io UI, and the listener host depends on our region’s account, for example, listener.logz.io or  listener-eu.logz.io.  

Once we have these two credentials, we’ll replace the placeholders in the following kubectl command and execute it:

kubectl create secret generic logzio-logs-secret \
--from-literal=logzio-log-shipping-token=<>' \
--from-literal=logzio-log-listener='https://<> \
-n kube-system

This message is displayed:

secret/logzio-logs-secret created

Next, we’ll deploy the dameonset with:

kubectl apply -f
https://raw.githubusercontent.com/logzio/logzio-k8s/master/logzio-daemonset-rbac.yaml

And the output:

serviceaccount/fluentd created
clusterrole.rbac.authorization.k8s.io/fluentd created
clusterrolebinding.rbac.authorization.k8s.io/fluentd created
daemonset.extensions/fluentd-logzio created

We can verify that the Logz.io fluentd pods are running with:

kubectl get pods -n kube-system grep | logzio

Here we see three pods, one per node:

fluentd-logzio-4bskq              1/1     Running     0     58s
fluentd-logzio-dwvmw              1/1     Running     0     58s
fluentd-logzio-gg9bv              1/1     Running     0     58s

And within a minute or two, we should see logs flowing into Logz.io from our Kubernetes cluster:

Logs flowing into Logz.io from a Kubernetes cluster

Step 3: Analyzing AKS logs in Logz.io

Great, we’ve built a logging pipeline from our Kubernetes cluster on AKS to Logz.io. What next? How do we make sense of all the log data being generated by our cluster?

Container logs are shipped in JSON format using Docker’s json-file logging driver. This means that they will be parsed automatically by Logz.io. This makes it much easier to slice and dice the data with the analysis tools provided by Logz.io. 

Still, some messages might require some extra parsing, in which case we would need to tweak the fluentd configuration or simply ping Logz.io’s support team for help.

We can query the logs using different queries. You could perform a simple free-text search looking for errors but Kibana offers much more advanced filtering and querying options that will help you find the information you’re looking for. 

For example, here we’re using the filter box to easily look at the logs generated by our frontend voting service:

You could perform a simple free-text search looking for errors in AKS logs but Kibana offers much more advanced filtering and querying options

Logz.io also provides advanced machine learning capabilities that help reveal events that otherwise would have gone unnoticed within the piles of log messages generated in our environment. 

In the example below, Cognitive Insights has flagged an issue with etcd, the Kubernetes key value store used as for storing cluster data. Opening the event reveals contextual information that helps us understand whether there is a real issue here or not:

Cognitive Insights has flags an issue with etcd, the Kubernetes key value store used as for storing cluster data

If you want to see a live feed of the cluster logs, either in their raw format or in parsed format, you can use Logz.io’s Live Tail page. Sure, you could use the kubectl logs command to tail logs but in an environment consisting of multiple nodes and an even larger amount of pods, this approach is far from being efficient. 

See a live feed of the cluster logs, either in their raw format or in parsed format, you using Logz.io’s Live Tail

Step 4: Building a monitoring dashboard

Kibana is a great tool for visualizing log data. You can create a variety of different visualizations that help you monitor your Kubernetes cluster — from simple metric visualizations to line charts and geographical maps. Below are a few basic examples.

Number of pods

Monitoring the number of pods running will show you if the number of nodes available is sufficient and if they will be able to handle the entire workload in case a node fails. A simple metric visualization can be created to help you keep tabs on this number:

Simple metric visualization in Kibana

Logs per pod 

Monitoring noisy pods or a sudden spike in logs from a specific pod can indicate whether an error taking place. You could create a bar chart visualization to monitor the log traffic:

Create a bar chart visualization to monitor the log traffic from AKS

Once you have all your visualizations lined up, you can add them to a dashboard that provides a comprehensive picture of how our cluster is performing.

Integrated Kibana dashboard for AKS logs shipped to Logz.io

Endnotes

Logging Kubernetes is challenging. There are multiple log types to make sense of and also a large volume of them to handle. These logs need to be collected, processed and stored. That’s where centralized logging systems like Logz.io come into the picture. 

The combination of AKS and the analysis tools provided in the ELK Stack and Logz.io is a powerful combination that can help simplify not only the deployment and management of your Kubernetes cluster but also troubleshooting and monitoring it.