#Docker
#Docker Monitoring
#Infrastructure Monitoring
#Metrics
#Prometheus
#Tutorials

An Open Source Prometheus Tutorial for System and Docker Monitoring

By: Gedalyah Reback
Prometheus Tutorial: Getting Started with Node Explorer and cAdvisor

Prometheus is an essential element in DevOps. It has one of the largest contributor communities in the open source world and is the favorite database for time-series metrics data. It is also part of Logz.io’s complete observability package, which integrates Prometheus, Jaeger, and ELK Stack into one platform.  It’s a favorite for system and container monitoring. This tutorial will show you how Prometheus works with Docker and your local machine.

Prometheus runs on and monitors many types of systems. Installation and configuration, therefore, can vary from software to software, platform to platform, and environment to environment. This tutorial is meant to be a basic introduction to setup on your local environment, on Docker, on Kubernetes, and through other means.

What is Prometheus?

Prometheus is an open-source monitoring system for processing time series metric data. Prometheus collects, organizes, and stores metrics using unique identifiers and timestamps. DevOps teams and developers query that data using the PromQL querying language and then visualize it in a UI such as Grafana.

Prometheus is reputable for many reasons, but an important one is its long list of easy integrations with other tools. In order to facilitate those connections, Prometheus uses exporters instead of agents to communicate. 

Prometheus Exporters

For every major DevOps tool on the market or emerging from Open Source, there is a specific exporter to move metrics into Prom. The most important of these is the Node_Exporter, to export system metrics. The full list of Prometheus exporters is long, but some critical examples include:

Available exporters

Alternatively, developers might choose to instrument code for Prometheus metric types. Prometheus maintains four official client libraries for the following languages: Go, Java / Scala, Python, and Ruby. But in addition, Prometheus users have created libs for the following languages:

Bash, C, C++, Common Lisp, Dart, Elixir, Erlang, Haskell, Lua for Nginx and Lua for Tarantool, .NET / C#, Node.js, Perl, PHP, R, Rust

Prometheus Installation (and Node Exporter)

From Precompiled Binary

Download the correct download package from Prometheus’ official site:

wget https://github.com/prometheus/prometheus/releases/download/v*/prometheus-*.*-amd64.tar.gz
tar xvf prometheus-*.*-amd64.tar.gz
cd prometheus-*.*

Then add prometheus to your terminal path.

Alternatively, install with Homebrew if you’re using a Mac: 

brew install prometheus

And to run prometheus you’ll need to run it with your configuration file:

prometheus --config.file=your_config.yml

From Source

Next you’ll have to create a Prometheus user (if you don’t have one already). From there, create a new directory, config file, and a few other things. First, start with the new directory:

sudo mkdir -p $GOPATH/src/github.com/prometheus
cd $GOPATH/src/github.com/prometheus
sudo git clone https://github.com/prometheus/prometheus.git
cd prometheus
make build

Next, create a new Prometheus configuration file using the touch or vi or vim commands:

sudo vi prometheus.yml

Prometheus & Docker

Once you have Docker configured, pull the official Prometheus Docker image: 

docker pull prom/prometheus

Then, prompt Docker to relocate it to the relevant container. Do this by running the prom/prometheus image mounted and bound. When you move the prometheus.yml config file, make sure you first mark the current location: 

$ docker run \
     -p 9090:9090 \
     -v /PATH/TO/prometheus.yml:/etc/prometheus/prometheus.yml \
     prom/prometheus

Now, to target Docker with Prometheus, edit the daemon.json file. In Linux, head to /etc/docker/daemon.json. In Docker Desktop on either Mac or Windows, follow this sequence:

Docker Icon → Preferences → Docker Engine

Then, paste the following into the configuration box, whether the box is empty or as an addition to what’s already:

{
  "metrics-addr" : "127.0.0.1:9323",
  "experimental" : true
}

Next, if you don’t already have one, create a prometheus.yml file.

Add the following basic configuration (from Prometheus) to that file:

#global config
global:
  scrape_interval:     15s
  evaluation_interval: 5s
  scrape_timeout:     1m
  #query_log_file: <string>
  external_labels:
    monitor: 'codelab-monitor'
# Scrape configs only contain one scrape target
scrape_configs:
  - job_name: 'prometheus'
    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
    static_configs:
      - targets: [docker.for.mac:9323']

Installing the Exporter for System and Docker

Install Node Exporter

wget https://github.com/prometheus/node_exporter/releases/download/v*/node_exporter-*.*-amd64.tar.gz
tar xvfz node_exporter-*.*-amd64.tar.gz
cd node_exporter-*.*-amd64
./node_exporter

Alternatively:

brew install node_exporter
brew services start node_exporter

Then reopen your prometheus.yml file and add the following:

scrape_configs:
- job_name: node
  static_configs:
  - targets: ['localhost:9100']

For Docker (cAdvisor)

Node Exporter is for exporting local system metrics. For Docker (and Kubernetes), you will need cAdvisor. Do not use Container Exporter; it’s been deprecated.

brew install docker-compose

In prometheus.yml, add the following. Obviously, you can alter the scrape interval:

scrape_configs:
- job_name: cadvisor
  scrape_interval: 5s
  static_configs:
  - targets:
    - cadvisor:8080

Then you will have to alter or create the docker-compose.yml config file:

version: '3.2'
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
    - 9090:9090
    command:
    - --config.file=/etc/prometheus/prometheus.yml
    volumes:
    - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    depends_on:
    - cadvisor
  cadvisor:
    image: gcr.io/google-containers/cadvisor:latest
    container_name: cadvisor
    ports:
    - 8080:8080
    volumes:
    - /:/rootfs:ro
    - /var/run:/var/run:rw
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro
    depends_on:
    - redis
  redis:
    image: redis:latest
    container_name: redis
    ports:
    - 6379:6379

#Note: A common error with applications whose default ports are 8080 (cAdvisor, Jenkins, JIRA, Apache Tomcat, and as an alt for HTTP) are is that attempts to turn on a container using that port will be met with a message like this:

Error response from daemon: Ports are not available: listen tcp 0.0.0.0:8080: bind: address already in use

You can get around this by directing it to an alternative port. Kill the current container for cAdvisor, then launch a new version:

docker container create -p 8081:8080 --name cadvisor-container google/cadvisor

At that point, you can check to see Node Exporter is running by going to localhost:9100/metrics, or looking at it in the terminal itself with the following command:

curl localhost:9100/metrics

The metrics there will be the same that Prometheus scrapes.

Your New Metric System

For more info on running Prometheus and Logz.io’s new Prometheus-as-a-service, check out our recent announcement. For further guidance on metrics, check out our Grafana tutorial.

Get started for free

Completely free for 14 days, no strings attached.
Start Free Trial
Schedule Demo