Hosted Kubernetes services such as AKS were introduced to help engineers deal with the complexity involved in deploying and managing Kubernetes clusters. They do not cover, however, the task of monitoring Kubernetes and the services running on it.
While some of the hosted Kubernetes services offer logging solutions, they do not offer all the functionality, flexibility and user-experience expected from a modern log management solution. The ELK Stack, or the EFK Stack to be more precise fills in these gaps by providing a Kubernetes-native logging experience — fluentd (running as a daemonset) to aggregate the different logs from the cluster, Elasticsearch to store the data and Kibana to slice and dice it.
This article will show how to hook up an AKS Kubernetes cluster with Logz.io’s ELK Stack. You’ll need two things for following the steps outlined below — an existing Kubernetes cluster on AKS and a Logz.io account.
Step 1: Deploy a demo application on AKS
To simulate some load and generate log data, we’ll deploy a demo application on our AKS cluster. For this purpose, we’ll use the Azure voting app which includes two Kubernetes Services – a Redis instance and an external service for accessing the app.
If you already have services running on AKS, feel free to skip directly to the next step.
So first, we’ll create a new manifest file containing the specs of deploying the app:
vim azure-vote.yaml
We’ll use this YAML file for deploying the app:
apiVersion: apps/v1
kind: Deployment
metadata:
name: azure-vote-back
spec:
replicas: 1
selector:
matchLabels:
app: azure-vote-back
template:
metadata:
labels:
app: azure-vote-back
spec:
nodeSelector:
"beta.kubernetes.io/os": linux
containers:
- name: azure-vote-back
image: redis
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 250m
memory: 256Mi
ports:
- containerPort: 6379
name: redis
---
apiVersion: v1
kind: Service
metadata:
name: azure-vote-back
spec:
ports:
- port: 6379
selector:
app: azure-vote-back
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: azure-vote-front
spec:
replicas: 1
selector:
matchLabels:
app: azure-vote-front
template:
metadata:
labels:
app: azure-vote-front
spec:
nodeSelector:
"beta.kubernetes.io/os": linux
containers:
- name: azure-vote-front
image: microsoft/azure-vote-front:v1
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 250m
memory: 256Mi
ports:
- containerPort: 80
env:
- name: REDIS
value: "azure-vote-back"
---
apiVersion: v1
kind: Service
metadata:
name: azure-vote-front
spec:
type: LoadBalancer
ports:
- port: 80
selector:
app: azure-vote-front
To deploy the app, we’ll use:
kubectl apply -f azure-vote.yaml
We’ll get the following output:
deployment "azure-vote-back" created service "azure-vote-back" created deployment "azure-vote-front" created service "azure-vote-front" created
To verify the app is working, we’re going to access it in our browser. As part of the deployment process, a Kubernetes service exposes the app’s front end to the internet. This might take a minute or two, and we can follow the status of the deployment with:
kubectl get service azure-vote-front --watch NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE azure-vote-front LoadBalancer 10.0.232.187 52.188.177.27 80:32515/TCP 1m
As soon as an external IP is available for the service, we’ll simply paste it in our browser:
As part of the deployment process, a Kubernetes service exposes the Azure app’s front end to the internet.
Step 2: Ship Kubernetes logs to Logz.io
Our cluster and the app deployed on it is now generating a mix of log data, all useful for gaining insight into how our environment is performing. Our next step is to ship this log data into Logz.io.
To do this, we’ll use a daemonset that runs a fluentd pod on each node in our Kubernetes cluster.
Our first step is to store Logz.io credentials as a Kubernetes secret — our Logz.io account’s shipping token and Logz.io’s listener host. We can find the shipping token in the Logz.io UI, and the listener host depends on our region’s account, for example, listener.logz.io or listener-eu.logz.io.
Once we have these two credentials, we’ll replace the placeholders in the following kubectl command and execute it:
kubectl create secret generic logzio-logs-secret \ --from-literal=logzio-log-shipping-token=<>' \ --from-literal=logzio-log-listener='https://<> \ -n kube-system
This message is displayed:
secret/logzio-logs-secret created
Next, we’ll deploy the dameonset with:
kubectl apply -f https://raw.githubusercontent.com/logzio/logzio-k8s/master/logzio-daemonset-rbac.yaml
And the output:
serviceaccount/fluentd created clusterrole.rbac.authorization.k8s.io/fluentd created clusterrolebinding.rbac.authorization.k8s.io/fluentd created daemonset.extensions/fluentd-logzio created
We can verify that the Logz.io fluentd pods are running with:
kubectl get pods -n kube-system grep | logzio
Here we see three pods, one per node:
fluentd-logzio-4bskq 1/1 Running 0 58s fluentd-logzio-dwvmw 1/1 Running 0 58s fluentd-logzio-gg9bv 1/1 Running 0 58s
And within a minute or two, we should see logs flowing into Logz.io from our Kubernetes cluster:
Logs flowing into Logz.io from a Kubernetes cluster
Step 3: Analyzing AKS logs in Logz.io
Great, we’ve built a logging pipeline from our Kubernetes cluster on AKS to Logz.io. What next? How do we make sense of all the log data being generated by our cluster?
Container logs are shipped in JSON format using Docker’s json-file logging driver. This means that they will be parsed automatically by Logz.io. This makes it much easier to slice and dice the data with the analysis tools provided by Logz.io.
Still, some messages might require some extra parsing, in which case we would need to tweak the fluentd configuration or simply ping Logz.io’s support team for help.
We can query the logs using different queries. You could perform a simple free-text search looking for errors but Kibana offers much more advanced filtering and querying options that will help you find the information you’re looking for.
For example, here we’re using the filter box to easily look at the logs generated by our frontend voting service:
You could perform a simple free-text search looking for errors in AKS logs but Kibana offers much more advanced filtering and querying options
Logz.io also provides advanced machine learning capabilities that help reveal events that otherwise would have gone unnoticed within the piles of log messages generated in our environment.
In the example below, Cognitive Insights has flagged an issue with etcd, the Kubernetes key value store used as for storing cluster data. Opening the event reveals contextual information that helps us understand whether there is a real issue here or not:
Cognitive Insights has flags an issue with etcd, the Kubernetes key value store used as for storing cluster data
If you want to see a live feed of the cluster logs, either in their raw format or in parsed format, you can use Logz.io’s Live Tail page. Sure, you could use the kubectl logs command to tail logs but in an environment consisting of multiple nodes and an even larger amount of pods, this approach is far from being efficient.
See a live feed of the cluster logs, either in their raw format or in parsed format, you using Logz.io’s Live Tail
Step 4: Building a monitoring dashboard
Kibana is a great tool for visualizing log data. You can create a variety of different visualizations that help you monitor your Kubernetes cluster — from simple metric visualizations to line charts and geographical maps. Below are a few basic examples.
Number of pods
Monitoring the number of pods running will show you if the number of nodes available is sufficient and if they will be able to handle the entire workload in case a node fails. A simple metric visualization can be created to help you keep tabs on this number:
Simple metric visualization in Kibana
Logs per pod
Monitoring noisy pods or a sudden spike in logs from a specific pod can indicate whether an error taking place. You could create a bar chart visualization to monitor the log traffic:
Create a bar chart visualization to monitor the log traffic from AKS
Once you have all your visualizations lined up, you can add them to a dashboard that provides a comprehensive picture of how our cluster is performing.
Integrated Kibana dashboard for AKS logs shipped to Logz.io
Endnotes
Logging Kubernetes is challenging. There are multiple log types to make sense of and also a large volume of them to handle. These logs need to be collected, processed and stored. That’s where centralized logging systems like Logz.io come into the picture.
The combination of AKS and the analysis tools provided in the ELK Stack and Logz.io is a powerful combination that can help simplify not only the deployment and management of your Kubernetes cluster but also troubleshooting and monitoring it.
