Yesterday, we announced the beta release of Logz.io Infrastructure Monitoring — our Grafana-based monitoring solution, and the planned release of a Jaeger-based tracing solution. These additions to our platform complement our ELK-based Log Management product, together constituting what is the world’s only open source-based observability platform for monitoring, troubleshooting and securing distributed cloud workloads.
This article will dive a bit deeper into the steps required to set up an observability stack for Kubernetes using Logz.io — collecting logs and metrics using Fluentd and Metricbeat and then correlating between them using Kibana and Grafana, all in Logz.io.
Step 1: Collecting Kubernetes signals
Kubernetes has gone a long way since first being introduced and now makes available a long list of data signals, primarily logs and metrics, that can be collected and used for observability. We go over the different log types in this article.
To collect these signals, we need to build some data pipelines. Logz.io provided easy daemonset-based integrations using open-source agents — Fluentd for logs and Metricbeat for metrics.
Let’s take a closer look.
Shipping Kubernetes logs with fluentd
For shipping our Kubernetes logs into Logz.io, we will use a daemonset that runs a Fluentd pod on every node in the cluster. The image used is pre-configured to collect all the logs from a Kubernetes cluster together with relevant metadata for better context. Another option is using Filebeat, and we’ll cover this option in a subsequent article.
The first step is to store Logz.io credentials as a Kubernetes secret — your Logz.io account’s shipping token and Logz.io’s listener host. You can find the shipping token in the Logz.io UI, and the listener host depends on your region’s account, for example, listener.logz.io or listener-eu.logz.io.
Once you have these two credentials, replace the placeholders in the following kubectl command and execute it:
kubectl create secret generic logzio-logs-secret \ --from-literal=logzio-log-shipping-token='<>' \ --from-literal=logzio-log-listener='<>' \ -n kube-system
You should see this message displayed:
Next, simply deploy the dameonset:
kubectl apply -f https://raw.githubusercontent.com/logzio/logzio-k8s/master/logzio-daemonset-rbac.yaml
And the output:
serviceaccount/fluentd created clusterrole.rbac.authorization.k8s.io/fluentd created clusterrolebinding.rbac.authorization.k8s.io/fluentd created daemonset.extensions/fluentd-logzio created
Check that the Logz.io fluentd pods are running with:
kubectl get pods -n
Here we see three pods, one per node:
fluentd-logzio-4bskq 1/1 Running 0 58s fluentd-logzio-dwvmw 1/1 Running 0 58s fluentd-logzio-gg9bv 1/1 Running 0 58s
Within a minute or two, you should see logs flowing into Logz.io from your Kubernetes cluster:
Shipping Kubernetes metrics with Metricbeat
For shipping Kubernetes metrics into Logz.io, we will again use a daemonset but this time one that runs Metricbeat.
First, you’ll need to have kube-state-metrics installed in your cluster. Instructions for this vary a bit depending on what Kubernetes deployment you’re using, but the commands below will help you install it on an Amazon EKS cluster.
Start by cloning the project:
git clone https://github.com/kubernetes/kube-state-metrics.git cd kube-state-metrics
You can now install with:
kubectl apply -f examples/standard
You should see the following output:
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created clusterrole.rbac.authorization.k8s.io/kube-state-metrics created deployment.apps/kube-state-metrics created serviceaccount/kube-state-metrics created service/kube-state-metrics created
Next, we’re going to again store Logz.io credentials as a Kubernetes secret, this time using the shipping token for your Logz.io metrics account:
kubectl –namespace=kube-system create secret generic logzio-metrics-secret –from-literal=logzio-metrics-shipping-token=<<SHIPPING-TOKEN>> –from-literal=logzio-metrics-listener-host=<<LISTENER-HOST>>
The next step is to save your cluster details as a Kubernetes secret. You’ll need the following cluster details:
Replace the placeholders in the command below with the cluster details you retrieved and execute:
kubectl --namespace=kube-system create secret generic cluster-details --from-literal=kube-state-metrics-namespace=<> --from-literal=kube-state-metrics-port=<> --from-literal=cluster-name=<> secret/cluster-details created
Last but not least, deploy the Metricbeat daemonset with:
kubectl --namespace=kube-system create -f https://raw.githubusercontent.com/logzio/logz-docs/master/shipping-config-samples/k8s-metricbeat.yml
You should see the following output:
configmap/logzio-cert created configmap/metricbeat-config created configmap/metricbeat-daemonset-modules created daemonset.extensions/metricbeat-new created configmap/metricbeat-deployment-modules created clusterrolebinding.rbac.authorization.k8s.io/metricbeat created clusterrole.rbac.authorization.k8s.io/metricbeat created serviceaccount/metricbeat created deployment.apps/metricbeat created
Make sure our Metricbeat pods are running:
kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE aws-node-94l7z 1/1 Running 0 32m aws-node-ssfz9 1/1 Running 0 32m aws-node-vnwxq 1/1 Running 0 32m coredns-6f647f5754-cmbjw 1/1 Running 0 38m coredns-6f647f5754-fqtrn 1/1 Running 0 38m kube-proxy-fq9mx 1/1 Running 0 32m kube-proxy-qvhgc 1/1 Running 0 32m kube-proxy-zr56c 1/1 Running 0 32m kube-state-metrics-5458dddb44-nhvsc 1/1 Running 0 22m metricbeat-586b769957-nthqk 1/1 Running 0 8m53s metricbeat-new-kzp2b 1/1 Running 0 8m53s metricbeat-new-l7w2r 1/1 Running 0 8m53s metricbeat-new-xph8r 1/1 Running 0 8m53s
Within a minute or two, your Kubernetes metrics will be shipped into Logz.io. It’s time to explore them using Grafana.
Step 2: Correlating between Kubernetes signals
Collecting logs and metrics from Kubernetes is not complicated. As shown above, there are a number of open source data shippers that do the job perfectly well. The challenge lies in being able to easily correlate between the signals. Grafana is a great analysis tool for metrics and Kibana is the de facto open source standard for investigating logs. Now, if only the two could be put together with proper correlation capabilities?
That’s exactly what Logz.io provides. Let’s see how.
Monitoring Kubernetes metrics with Grafana
Your metrics are stored in Logz.io in dedicated Elasticsearch indices that were designed to provide you with an optimized analysis experience. Using Elasticsearch’s rollup feature, metrics are automatically aggregated and downsampled. The metrics themselves can be visualized using Grafana, under the Metrics page in Logz.io.
Logz.io provides multiple dashboards for monitoring Kubenetes both on the cluster level, the node level and the application level, as well as other dashboards for other environments. These can be found in the Logz.io Dashboards folder:
To start monitoring our Kubernetes cluster, I’m going to open the K8S Cluster dashboard:
Scrolling down, we see that we have an nginx pod that seems to be consuming a suspiciously high amount of memory and CPU:
To get a closer look, I’m going to filter the Pod Memory usage panel by clicking the nginx pod:
But this is just a symptom of an underlying issue. To drill down to the root cause, we’ll need to take a look at our nginx container logs. To this end, we’ll use the Explore in Kibana button on the panel.
Kibana is opened with all the relevant filters so you can seamlessly transition from monitoring metrics to log analysis. We can see that there’s a Java stack trace exception that’s showing up in the logs – which is probably the event to drill-down into.
Summing it up
A lot of engineers prefer to use open source observability tools. These tools offer flexibility, easy setup and community support, and of course, help reduce the dreaded vendor lock-in. For Kubernetes, these engineers will use fluentd to ship cluster and container logs into Elasticsearch, Prometheus and Grafana to build beautiful monitoring dashboards, and Jaeger to track performance issues across services.
Sounds great, right?
But there is an accompanying cost. At scale, the total cost of ownership rises. Engineers will have to spend time and resources to maintain these observability tools. Not only that, they will be less effective due to the siloed nature of these tools. When pressed for time, say when troubleshooting a production issue, for example, the last thing you want to bother yourself with is moving back and forth between different monitoring interfaces in different tools. It’s time-consuming and ineffective.
Logz.io aims to enable engineers to be more productive by allowing them to focus on what they do best and what matters most to their business — build and deliver great software. We do this by providing them with the best of breed open source observability tools they prefer to use as a fully managed solution, bundled in one unified platform and enhanced with enterprise-grade capabilities.
Give it a try. We’d love to get your feedback!