A Metricbeat Tutorial: Getting Started

The ELK Stack, traditionally comprised of Elasticsearch, Logstash, and Kibana, now also includes a fourth element — Beats, a family of log shippers for different use cases and sets of data. This tutorial covers Metricbeat, the metrics light shipper companion to Filebeat, which is for logs.

Beats include a variety of different log shippers. In a previous tutorial, we described how to work with Filebeat for shipping log files into the stack.

What is Metricbeat?

Metricbeat evolved out of Topbeat and like the other beats, is built upon Libbeat — a Go framework. As its name implies, Metricbeat is a lightweight shipper that collects and ships various system and service metrics to a specified output destination.

Metricbeat is installed on the different servers in your environment and used for monitoring their performance, as well as that of different external services running on them. For example, you can use Metricbeat to monitor and analyze system CPU, memory and load. In Dockerized environments, Metricbeat can be installed on a host for monitoring container performance metrics.

Metricbeat is usually configured to ship the data directly to an Elasticsearch deployment. To gain architectural flexibility though, Metricbeat can also be configured to ship to a messaging or queuing buffer such as Kafka or Redis.

Installing Metricbeat

Metricbeat can be downloaded and installed using a variety of different methods and on a variety of supported platforms. Of course, you will need to have a running ELK Stack to be able to ship and store the collected metrics. I will outline two methods, using Apt and Docker, but you can refer to the official docs for more options.

Install Metricbeat on Linux using Apt

For an easier way of updating to a newer version, and depending on your Linux distro, you can use Apt or Yum to install Metricbeat from Elastic’s repositories:

First, you need to add Elastic’s signing key so that the downloaded package can be verified (skip this step if you’ve already installed packages from Elastic):

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key 
add -

The next step is to add the repository definition to your system:

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo 
tee -a /etc/apt/sources.list.d/elastic-7.x.list

All that’s left to do is to update your repositories and install Metricbeat:

sudo apt-get update && sudo apt-get install metricbeat

Install Metricbeat using Homebrew

brew install elastic/tap/metricbeat-full

To install the -oss version, enter the following:

brew install elastic/tap/metricbeat-oss

or

brew install metricbeat

Install Metricbeat on Docker

If you’re running Docker, you can install Metricbeat as a container on your host and configure it to collect either host metrics or container metrics.

To install Metricbeat as a container, pull Elastic’s Metricbeat image:

docker pull docker.elastic.co/beats/metricbeat:7.4.2

Install Metricbeat on Kubernetes

You can also deploy a Docker image of Metricbeat on Kubernetes.

curl -L -O https://raw.githubusercontent.com/elastic/beats/7.6/deploy/kubernetes/metricbeat-kubernetes.yaml

Configuring Metricbeat

Before we run Metricbeat, we need to understand the basics of how to configure it, first and foremost what configuration options are available within the main configuration file.

Metricbeat Configuration File

Metricbeat is configured using a YAML configuration file. On Linux, this file is located at: /etc/metricbeat/metricbeat.yml.

On Docker, you will find it at: /usr/share/metricbeat/metricbeat.yml.

If you are using Homebrew, it will be located at usr/local/etc/metricbeat/metricbeat.yml or usr/local/etc/metricbeat/metricbeat.yml.default.

YAML is syntax sensitive. You cannot, for example, use tabs for spacing. There are a number of additional best practices that will help you avoid mistakes in this Musing in YAML article.

Metricbeat Modules and Metricsets

Metricbeat modules contain service-specific collection and connection definitions. They define what specific metrics to collect from the service, the frequency in which to collect them, and how to connect to it. Modules, in turn, are comprised of one or multiple metricsets – literally, a set of related metrics to be collected and shipped.

Metricbeat supports a growing number of modules, such as System, Apache, nginx, MySQL and Docker, to name just the popular and more commonly used modules. By default, Metricbeat is configured to use the system module which collects server system metrics, such as CPU and memory usage, network, and so forth.

Here is an example of a Metricbeat configuration file that has two modules defined, the System and Apache modules:

metricbeat.modules:
- module: system
  metricsets: ["cpu","memory","network"]
  enabled: true
  period: 15s
  processes: ['.*']
- module: apache
  metricsets: ["status"]
  enabled: true
  period: 5s
  hosts: ["http://172.20.11.7"]

The system module may also appear as such in the config file:

- module: system
  period: 10s
  metricsets:
    - cpu
    - load
    - memory
    - network
    - process
    - process_summary
    - socket_summary
    #- entropy
    #- core
    #- diskio
    #- socket
  process.include_top_n:
    by_cpu: 5      # include top 5 processes by CPU
    by_memory: 5   # include top 5 processes by memory
- module: system
  period: 1m
  metricsets:
    - filesystem
    - fsstat
  processors:
  - drop_event.when.regexp:
      system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
- module: system
  period: 15m
  metricsets:
    - uptime

Note

If you are using MacOS or another POSIX-compliant operating system, keep in mind that all Beats config files (including Metricbeat configuration files) have permission checks. So in order to start Metricbeat, you have to use the chown root command (i.e., you must assert dominance).

sudo chown root metricbeat.yml

sudo chown root modules.d/system.yml

At this point, enter the following commands to make sure Metricbeat is running. If successful, the messages below those commands should appear.

$ sudo metricbeat modules enable system

Index setup finished.

$ sudo service metricbeat start

Loaded dashboards

Metricbeat configuration settings

For each module, define a list of module-specific metricsets. We cannot define a metricset that is not supported by the module used.

Using the period setting, we define the frequency at which we want Metricbeat to fetch the metrics.

The hosts setting is optional, and it defines the host, or hosts, we want Metricset to fetch the metrics from. You can also use fields and tags to add custom fields and tags to the metricset events.

Another optional setting is processors, which can be used to apply different changes to the data collected by Metricbeat. For example, you can use processors to drop specific fields, drop specific events, add metadata and more.

There are two main methods for enabling modules – you can enable them in the modules.d directory in which they are installed (/etc/metricbeat/modules.d) or as shown in the example above from within the configuration file. (enabled: true).

Configuring Metricbeat to Monitor Docker

Using the Docker module, Metricbeat can be configured to ship a bunch of useful information about Docker containers running on the host:

<!-- wp:preformatted {"className":"lang:default decode:true"} -->
<pre class="wp-block-preformatted lang:default decode:true"></pre>
<!-- /wp:preformatted -->

Configuring Metricbeat in Kubernetes

If you want to redirect metrics to another place besides your standing deployment of Elasticsearch, change the following fields in the manifest file where needed:

- name: ELASTICSEARCH_HOST
value: elasticsearch
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: elastic
- name: ELASTICSEARCH_PASSWORD
value: changeme


Metricbeat Output

This section in the Metricbeat configuration file defines where you want to ship the metrics to. As we specified above, in most cases you will make do with shipping the events directly into Elasticsearch, but Metricbeat can be configured to ship the data to other destinations.

For more advanced processing and data enhancement, you can define Logstash as your output (remember to use the beats input plugin in Logstash to collect the data from metricbeat). Kafka and Redis also have support as output destinations, for more complex and traffic-heavy pipelines.

Output within the ELK Stack

For forwarding metrics to Elasticsearch:

output.elasticsearch:
  hosts: ["localhost:9200"]

For forwarding metrics to Logstash:

output.logstash:
  hosts: ["localhost:5044"]

Output to Message Queues/Log Aggregators

For forwarding metrics to Kafka:

output.kafka:
  hosts: ["localhost:9092"]
  topic: "topicname"

To add topic selector rules, use the topics setting. In this example, error and access logs are forwarded to separate topics when identified by certain text in respective logs:

output.kafka:
  hosts: ["localhost:9092"]
  topic: "topicname"

For forwarding metrics to Redis:

output.redis:
  hosts: ["localhost:6379"]
  index: "redis.index.for.logstash"
  key: "metricbeat.to.redis" #If you don't designate a specific key, this will default to the specified index

Running Metricbeat

Now that we understand the basics of configuring Metricbeat, it’s time to actually run the program.

Before we do so, however, it’s a good practice to test your configuration file. To do this, access the directory where Metricbeat is installed, and run Metricbeat with the test subcommand:

cd /usr/share/metricbeat
sudo metricbeat -c /etc/metricbeat/metricbeat.yml test config
config OK

Depending on how you installed Metricbeat, enter the following commands to start Metricbeat.

Apt

Start the Metricbeat service with:

sudo service metricbeat start

Homebrew

brew services start elastic/tap/metricbeat-full

or

brew services start metricbeat

Docker

Run the Metricbeat container with defining bind-mounting to your configuration file (you can of course do the same thing by building your own image from a Dockerfile and running it). Be sure you have the correct permissions to connect to the Docker daemon:

sudo docker run -v 
/etc/metricbeat/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml 
docker.elastic.co/beats/metricbeat:7.4.2

If all runs as expected, you should be seeing a new index generate in Elasticsearch:

curl -XGET 'localhost:9200/_cat/indices?v&pretty'

health status index                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   metricbeat-7.4.2-2019.12.01 YePjpOgqQR-pU_SuKrm5vw   1   1       2989            0      1.1mb          1.1mb

Define the new index pattern in Kibana to start the analysis.

Shipping to Logz.io

By making a few adjustments to your Metricbeat configuration file, you can send metrics to the Logz.io ELK Stack for analysis and visualization.

First, you will need to download an SSL certificate to use encryption:

wget https://raw.githubusercontent.com/logzio/public-certificates/master/COMODORSADomainValidationSecureServerCA.crt

sudo mkdir -p /etc/pki/tls/certs

sudo cp COMODORSADomainValidationSecureServerCA.crt /etc/pki/tls/certs/

Next, retrieve your Logz.io account token from the UI (under Settings → Manage Accounts).

Finally, to ship the data to Logz.io, your Metricbeat configuration file needs to look as follows:

metricbeat.modules:
- module: system
  metricsets: ["cpu","memory","network"]
  enabled: true
  period: 15s
  processes: ['.*']

fields:
  logzio_codec: json
  token: <yourToken>
fields_under_root: true
ignore_older: 3hr
type: system_metrics

output:
  logstash:
    hosts: ["listener.logz.io:5015"]     
    ssl:
      certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']

Logz.io provides ready-made dashboards for monitoring the data that Metricbeat shipped. To install it, simply open ELK Apps, Logz.io’s library of pre-made dashboards and visualizations and search for ‘metricbeat’.

In this case, we installed the Docker Monitoring dashboard:

What next?

Metricbeat is an extremely easy-to-use, efficient and reliable metric shipper for monitoring your system and the processes running on it. Like the other beats in the family – Filebeat, Winlogbeat, Packetbeat, Auditbeat and Heartbeat– Metricbeat leaves a light resource footprint and can be installed and started relatively quickly.

Like in any software, there are a some quirks, mainly related to issues with configuring  YAML syntax. Tips on configuring YAML configuration files can be found in the Musings in YAML article.

Your next step is of course to analyze and visualize the data in Kibana. Take a look at our Kibana tutorials (Part 1 – Searching, Part 2 – Visualizing) for more information on this.

Also take a look at our Filebeat tutorial.