Deploying the ELK Stack on Kubernetes with Helm

By: Daniel Berman

Helm and Kubernetes are quick becoming a standard pairing. ELK and Kubernetes are used in the same sentence usually in the context of describing a monitoring stack. ELK integrates natively with Kubernetes and is a popular open-source solution for collecting, storing and analyzing Kubernetes telemetry data. However, ELK and Kubernetes are increasingly being used in another context—a method for deploying and managing the former. While deploying the ELK Stack using Kubernetes might seem like a complex task, there are more and more best practices around this scenario as well as Kubernetes-native solutions. One of these solutions is using Helm charts.

What’s Helm?

Maintained by CNCF, Helm is increasingly becoming a standard way for managing applications on Kubernetes. The easiest way to think about Helm is as a package manager for Kubernetes. It’s actually a bit more than just a package manager though as it allows users to create, publish and share applications on Kubernetes.

Each Helm chart contains all the specifications needed to be deployed on Kubernetes in the form of files describing a set of Kubernetes resources and configurations. Charts can be used to deploy very basic applications but also more complex systems such as…the ELK Stack!

Earlier this year, the folks at Elastic published Helm charts for Elasticsearch, Kibana, Filebeat and Metricbeat, making the deployment of these components on Kubernetes extremely simple.

Let’s take a closer look.

Setup

For the sake of this tutorial, I used Minikube installed on my Mac. You’ll also need Kubectl set up and configured.

Step 1: Setting Up Kubernetes for ELK Stack

Obviously, we first need to make sure we have a Kubernetes cluster to install the ELK Stack on.

When starting Minikube, you’ll need to allocate some extra firepower as the plan is to deploy a multi-node Elasticsearch cluster:

minikube start --cpus 4 --memory 8192

You should see output similar to this:

Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.

Just to verify your single-node Kubernetes cluster is up and running, use:

kubectl cluster-info

Kubernetes master is running at https://192.168.99.106:8443
KubeDNS is running at https://192.168.99.106:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

Step 2: Installing Helm on Kubernetes

Your next step is to install Helm. Again, if you’ve got Helm setup and initialized already, great, you can skip to deploying the ELK Stack in the following steps.

To install Helm. execute the following 3 commands:

curl https://raw.githubusercontent.com/kubernetes/Helm/master/scripts/get > get_Helm.sh
chmod 700 get_Helm.sh
./get_Helm.sh

You should see the following output:

Downloading https://get.Helm.sh/Helm-v2.14.3-darwin-amd64.tar.gz
Preparing to install Helm and tiller into /usr/local/bin
Helm installed into /usr/local/bin/Helm
tiller installed into /usr/local/bin/tiller
Run 'Helm init' to configure Helm.

To start Helm, enter:

Helm init

To verify the Tiller server is running properly, use:

kubectl get pods -n kube-system | grep tiller

And the output:

tiller-deploy-77b79fcbfc-hmqj8 1/1 Running 0 50s

Step 3: Deploying an Elasticsearch Cluster with Helm

It’s time to start deploying the different components of the ELK Stack. Let’s start with Elasticsearch.

As mentioned above, we’ll be using Elastic’s Helm repository so let’s start with adding it:

Helm repo add elastic https://Helm.elastic.co
"elastic" has been added to your repositories

Next, download the Helm configuration for installing a multi-node Elasticsearch cluster on Minikube:

curl -O https://raw.githubusercontent.com/elastic/Helm-charts/master/elasticsearch/examples/minikube/values.yaml

Install the Elasticsearch Helm chart using the configuration you just downloaded:

Helm install --name elasticsearch elastic/elasticsearch -f ./values.yaml

The output you should be seeing looks something like this:

NAME:   elasticsearch
LAST DEPLOYED: Mon Sep 16 17:28:20 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME                    READY  STATUS   RESTARTS  AGE
elasticsearch-master-0  0/1    Pending  0         0s

==> v1/Service
NAME                           TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)            AGE
elasticsearch-master           ClusterIP  10.101.239.94         9200/TCP,9300/TCP  0s
elasticsearch-master-headless  ClusterIP  None                  9200/TCP,9300/TCP  0s

==> v1beta1/PodDisruptionBudget
NAME                      MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
elasticsearch-master-pdb  N/A            1                0                    0s

==> v1beta1/StatefulSet
NAME                  READY  AGE
elasticsearch-master  0/3    0s


NOTES:
1. Watch all cluster members come up.
  $ kubectl get pods --namespace=default -l app=elasticsearch-master -w
2. Test cluster health using Helm test.
  $ Helm test elasticsearch

As noted at the end of the output, you can verify your Elasticsearch pods status with:

kubectl get pods --namespace=default -l app=elasticsearch-master -w

It might take a minute or two, but eventually, three Elasticsearch pods will be shown as running:

NAME                     READY     STATUS     RESTARTS   AGE
elasticsearch-master-0   1/1       Running   0         1m
elasticsearch-master-2   1/1       Running   0         1m
elasticsearch-master-1   1/1       Running   0         1m

Our last step for deploying Elasticsearch is to set up port forwarding:

kubectl port-forward svc/elasticsearch-master 9200

Advanced Elasticsearch Configurations with Helm Charts

There is support for loadBalancerSourceRanges, which specifies exceptions of ranges of IP addresses that can access the designated load balancer. This is also available for Kibana.

Step 4: Deploying Kibana with Helm

Next up — Kibana. As before, we’re going to use Elastic’s Helm chart for Kibana:

Helm install --name kibana elastic/kibana

And the output:

NAME:   kibana
LAST DEPLOYED: Wed Sep 18 09:52:21 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Deployment
NAME           READY  UP-TO-DATE  AVAILABLE  AGE
kibana-kibana  0/1    1           0          0s

==> v1/Pod(related)
NAME                            READY  STATUS             RESTARTS  AGE
kibana-kibana-6d7466b9b9-fbmsz  0/1    ContainerCreating  0         0s

==> v1/Service
NAME           TYPE       CLUSTER-IP    EXTERNAL-IP  PORT(S)   AGE
kibana-kibana  ClusterIP  10.96.37.129         5601/TCP  0s

Verify your Kibana pod is running (it might take a minute or two until the status turns to “Running”):

kubectl get pods

NAME                             READY     STATUS    RESTARTS   AGE
elasticsearch-master-0           1/1       Running   0          15m
elasticsearch-master-1           1/1       Running   0          15m
elasticsearch-master-2           1/1       Running   0          15m
kibana-kibana-6d7466b9b9-fbmsz   1/1       Running   0          2m

And last but not least, set up port forwarding for Kibana with:

kubectl port-forward deployment/kibana-kibana 5601

You can now access Kibana from your browser at: http://localhost:5601:

Advanced Kibana Configurations with Helm Charts

Customizable labels are available for pods and can be added to services. Pod labeling is also available here for Elasticsearch.

Step 5: Deploying Metricbeat with Helm

To set up a data pipeline, we’re going to end this tutorial with deploying the Metricbeat Helm chart:

Helm install --name metricbeat elastic/metricbeat

Within a minute or two, your Kubernetes cluster will display Metricbeat pods running alongside your Elasticsearch and Kibana pods:

kubectl get pods

NAME                                             READY     STATUS    RESTARTS   AGE
elasticsearch-master-0                           1/1       Running   0          11m
elasticsearch-master-1                           1/1       Running   0          11m
elasticsearch-master-2                           1/1       Running   0          11m
kibana-kibana-6d7466b9b9-bsfd5                   1/1       Running   0          6m
metricbeat-kube-state-metrics-bd55f95cc-8654c    1/1       Running   0          1m
metricbeat-metricbeat-kjj6z                      1/1       Running   0          1m
metricbeat-metricbeat-metrics-699db67c5c-b2fzs   1/1       Running   0          1m

If you curl Elasticsearch, you’ll see that metrics have already begun to be indexed in Elasticsearch:

curl localhost:9200/_cat/indices

green open .kibana_task_manager               QxPJtK5rQtGGguLRv5h9OQ 1 1   2 4 87.7kb  44.8kb
green open metricbeat-7.3.0-2019.09.18-000001 DeXaNAnMTWiwrQKNHSL0FQ 1 1 291 0  1.1mb 544.1kb
green open .kibana_1                          gk0OHIZDQWCNcjgb-uCBeg 1 1   4 0 30.3kb  15.1kb

All that’s left to do now is define the index pattern in Kibana and begin analyzing your data. In Kibana, go to the Management → Kibana → Index Patterns page, and click Create index pattern. Kibana will automatically identify and display the Metricbeat index:

Enter ‘metricbeat-*’ and on the next step select the @timestamp field to finalize the creation of the index pattern in Kibana.

Hop on over to the Discover page. You’ll see all the metrics being collected from your Kubernetes cluster by Metricbeat displayed:

Endnotes

These Helm charts are a great way to get started with ELK on Kubernetes but will require tweaking to be able to handle large payloads. Maintaining an ELK Stack in production is not an easy task to start out with and managing a multi-node, large Elasticsearch cluster on Kubernetes will require both engineering resources and strong infrastructure. I expect that as Helm becomes the standard way to build and deploy applications on Kubernetes, best practices will emerge for handling large scale ELK deployments as well. Looking forward to it!