Driving Manual or Automatic: Docker Hub vs. Helm for Deploying Prometheus on Kubernetes

While the introduction of new abstraction layers has made creating and designing distributed systems a complex task, it hasn’t prevented an increase in their popularity. One of the tools in use for orchestrating container-based distributed systems is Kubernetes. One of the most effective Kubernetes monitoring options is Prometheus, a leading open-source monitoring solution for container environments. Prometheus, which ex-Googlers created at SoundCloud, the second project the Cloud Native Computing Foundation (CNCF) accepted, after Kubernetes itself.

Some of Prometheus’ most important features are:

• Its multidimensional data model with support from its time-series database
• Its own query language, PromQL
• Support for scraping metric data in pull mode, as well as the traditional push mode
• Automatic service discovery (i.e. finding the targets of metric data)
• Multiple integrations, such as with Kubernetes and other CNCF open source projects

Once you’ve decided to implement Kubernetes, your next step is to decide how best to deploy Prometheus and related applications—such as Grafana—to the cluster. There are a few common patterns developers use to do this. This article will pit two of these patterns against each other: the manual way with kubectl and Docker Hub vs. the automated way with Helm.

Deploying Applications to Kubernetes

With creation of a deployment, the Kubernetes master schedules application instances that are defined in the deployment manifest. After creating the instances, deployment controllers start to monitor them and reschedule them if necessary (e.g., when a node goes down or is cordoned), providing a self-healing mechanism for the deployed applications.

Many approaches are popular in deploying applications to Kubernetes, including:

• YAML manifests
• Packing managers like Helm, CNAB, and Cloudsmith
• kubectl

This tutorial will compare the “do it yourself” approach using Docker Hub, static manifests and kubectl to the orchestrated approach with Helm and predefined charts.

Creating a Prometheus Deployment Manually with Docker Hub and kubectl

In this section, we’ll cover deployment to a Kubernetes cluster with a static YAML manifest and Docker Hub images. YAML is a text format suitable for humans to specify configuration. In Kubernetes, YAML is used to define Kubernetes resources.

Before getting started, you’ll need to have a running cluster with kubectl configured. In addition, you should create a separate namespace just for monitoring to make it easier to manage the monitoring stack. To create a namespace, run the command below:

kubectl create namespace monitoring

Prometheus needs access to resources’ metrics to get them from the Kubernetes control plane. For this, create a ClusterRole and ClusterRoleBinding by writing the following configuration to the file named role-config.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 name: prometheus
rules:
- apiGroups: [""]
 resources:
 - endpoints
 - nodes
 - nodes/proxy
 - pods
 - services
 verbs: ["get", "list", "watch"]
- apiGroups:
 - extensions
 resources:
 - ingresses
 verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
 verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: prometheus
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: prometheus
subjects:
- kind: ServiceAccount
 name: default
 namespace: monitoring

Then, run the following command:

kubectl apply -f role-config.yaml

The output should include:

clusterrole.rbac.authorization.k8s.io/prometheus configured
clusterrolebinding.rbac.authorization.k8s.io/prometheus configuredclusterrole.rbac.authorization.k8s.io/prometheus configured
clusterrolebinding.rbac.authorization.k8s.io/prometheus configured

Configuring Prometheus with Docker Hub on Kubernetes

Next, configure Prometheus. Create a ConfigMap that will mount inside the Prometheus container. Write the following to configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
 name: prometheus-server-conf
 labels:
   name: prometheus-server-conf
 namespace: monitoring
data:
 prometheus.yml: |-
   global:
     scrape_interval: 5s
     evaluation_interval: 5s
   scrape_configs:
     - job_name: 'kubernetes-apiservers'
       kubernetes_sd_configs:
       - role: endpoints
       scheme: https
       tls_config:
         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
       relabel_configs:
       - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
         action: keep
         regex: default;kubernetes;https
     - job_name: 'kubernetes-nodes'
       scheme: https
       tls_config:
         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
       kubernetes_sd_configs:
       - role: node
       relabel_configs:
       - action: labelmap
         regex: __meta_kubernetes_node_label_(.+)
       - target_label: __address__
         replacement: kubernetes.default.svc:443
       - source_labels: [__meta_kubernetes_node_name]
         regex: (.+)
         target_label: __metrics_path__
         replacement: /api/v1/nodes/${1}/proxy/metrics
     - job_name: 'kubernetes-pods'
       kubernetes_sd_configs:
       - role: pod
       relabel_configs:
       - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
         action: keep
         regex: true
       - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
         action: replace
         target_label: __metrics_path__
         regex: (.+)
       - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
         action: replace
         regex: ([^:]+)(?::\d+)?;(\d+)
         replacement: $1:$2
         target_label: __address__
       - action: labelmap
         regex: __meta_kubernetes_pod_label_(.+)
       - source_labels: [__meta_kubernetes_namespace]
         action: replace
         target_label: kubernetes_namespace
       - source_labels: [__meta_kubernetes_pod_name]
         action: replace
         target_label: kubernetes_pod_name
     - job_name: 'kubernetes-cadvisor'
       scheme: https
       tls_config:
         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
       kubernetes_sd_configs:
       - role: node
         relabel_configs:
       - action: labelmap
         regex: __meta_kubernetes_node_label_(.+)
       - target_label: __address__
         replacement: kubernetes.default.svc:443
       - source_labels: [__meta_kubernetes_node_name]
         regex: (.+)
         target_label: __metrics_path__
         replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
      - job_name: 'kubernetes-service-endpoints'
        kubernetes_sd_configs:
        - role: endpoints
        relabel_configs:
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
          action: replace
          target_label: __scheme__
          regex: (https?)
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
          action: replace
          target_label: __address__
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_service_name]
          action: replace
          target_label: kubernetes_name

This ConfigMap is created in the “monitoring” namespace. It includes several job descriptions defining which metrics to scrape. Kubernetes-apiserver pulls data from the API servers, node metrics with kubernetes-nodes, and metrics for cAdvisor with kubernetes-cadvisor.

Scraping Services and Pods with Prometheus

To enable scraping from services and pods, add the annotations prometheus.io/scrape and prometheus.io/port to the metadata section in their definitions.

Now that you have a ConfigMap configuration ready, apply it to the cluster with:

kubectl apply -f configmap.yaml

Create a deployment to read the metrics by generating a new file deployment.yaml and use the following configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: prometheus-deployment
 namespace: monitoring
spec:
 replicas: 1
 template:
   metadata:
     labels:
       app: prometheus-server
   spec:
     containers:
     - name: prometheus
       image: prom/prometheus:v2.12.0
       args:
         - "--config.file=/etc/prometheus/prometheus.yml"
         - "--storage.tsdb.path=/prometheus/"
       ports:
         - containerPort: 9090
       volumeMounts:
         - name: prometheus-config-volume
           mountPath: /etc/prometheus/
         - name: prometheus-storage-volume
           mountPath: /prometheus/
     volumes:
       - name: prometheus-config-volume
         configMap:
           defaultMode: 420
           name: prometheus-server-conf
       - name: prometheus-storage-volume
         emptyDir: {}

You may be wondering what all of this has to do with Docker Hub. The Docker Hub image sees its use as part of the configuration. It’s under the spec.template.spec.containers.image.

When the pods are running, you should be able to access the Prometheus dashboard after proxying the application to your local machine using the following:

kubectl port-forward $(kubectl get pods -n monitoring --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}') -n monitoring 8080:9090

Open a new browser window, and go to http://localhost:8080. You should now see a Prometheus home page.

Adding Alerts and More Metrics

Keep in mind that this process does not establish a complete monitoring solution. There’s no alerting set up, there are no useful dashboards yet, and many of the metrics are still missing. More configuration needs to be done before you can be confident that your monitoring stack is ready for production usage.

Deploying Prometheus with YAML manifests and Docker Hub images is time-consuming. The upside of the process is that it allows you to have total control over the configuration.

Every service, deployment, and StatefulSet is under your direct influence. In contrast, with Helm charts you can only manipulate a few settings exposed in a values.yaml file.

If you don’t rely much on community components you would need your own YAML anyway. In that case, Helm’s value would be less significant. Some also prefer to keep it simple and avoid dealing with Helm’s own stack; primarily with Helm’s server-side—Tiller—and some of its complexities around HA and RBAC, just to name a few.

Nonetheless, many choose to go with higher-level automation and abstraction, so opt for Helm. We will cover that in the next section.

Creating a Prometheus Deployment with Helm

Helm is the most popular package manager users employ with Kubernetes, and is part of the CNCF, together with Kubernetes and Prometheus. Others, such as Cloudsmith and Cloud Native Application Bundles (CNAB), aren’t as popular.

Helm Charts with Kubernetes and Prometheus

Helm’s big advantage is a unique packaging format called Chart, a collection of files that describe Kubernetes resources. A single chart can deploy a single simple resource, like a pod or a cluster rule.

However, charts usually deploy sets of microservices, whole application stacks, and cluster configurations as one manageable deployment. Helm uses a powerful templating engine that lets it control flow, environment variables, and descriptions.

Of course, Prometheus and Grafana aren’t the only products that benefit from charts. For another deployment example, check out this tutorial to deploy the ELK Stack on Kubernetes with Helm.

Helm Chart Repositories

Helm also uses chart repositories, which are basically HTTP servers that store curated and managed charts. Repositories can be 1) public (whereas the community maintains them) or 2) private and self-hosted. Chart repositories allow you to install and implement an entire preconfigured monitoring stack (like Prometheus) with just a few commands.

We’ll skip the Helm installation and configuration process here. If you’re interested in it, consult the official documentation.

To add a repository and install Prometheus Operator, use the following commands:

helm repo add stable https://kubernetes-charts.storage.googleapis.com/
helm install stable/prometheus-operator --name-template po

The Prometheus Operator repository consists of charts with the following services:

• Prometheus-operator
• Prometheus
• Node-exporter
• Kube-state-metrics and service monitors
• Grafana with dashboards and alerts

To view Grafana in your browser, start a port-forwarding using these commands:

kubectl port-forward $(kubectl get pods -l=app.kubernetes.io/name=grafana -o=jsonpath='{.items[*].metadata.name}') 3000:3000

Grafana should be visible at http://localhost:3000. You can log in using admin as the user name and prom-operator or admin as the password.

For more info on this topic, check out our Best Practices for Monitoring Kubernetes using Grafana. Also see our tutorial on deploying InfluxDB and Grafana with Helm to K8s.

Instead of spending hours writing YAML manifests, choosing the right Docker Hub image, and fixing bugs, you can install a powerful monitoring stack with just a few commands. Thanks to templates, you still have some control over configuration.

Why Isn’t Helm’s Charts Repository Perfect?

Using public repositories is tempting, especially when deadlines are looming. But, in reality, you have no control over their configurations or resources. What if you miss a vulnerability, or one of the chart authors maliciously planted a hidden threat?

Consider this worst-case scenario: What if a component’s failure stops the production cluster from working?

With limited knowledge of what actually applied to the cluster, debugging the platform during an outage would be a nightmare. To prevent it, you may consider using Helm with a private container registry with a curated list of home-grown charts alongside gated and approved community charts. This way, you get the benefits of community contributed content without the risks involved with losing control over the components.

Summary

This article covered the two main approaches to configuring Prometheus deployments on Kubernetes: manual deployment of static YAML manifests with kubectl and Docker Hub images, versus automated and dynamic deployment with Helm charts.

Which one is the better choice? As is often the case, it really depends on your desired outcome.

Creating your own manifests from Docker Hub images gives you greater control over what is actually deployed inside the Kubernetes cluster. Using predefined Helm charts from stable repositories is incomparably quicker and error-proof. Either way, you end up with a solution that you’ll have to maintain, so make sure you feel comfortable with it.