Infrastructure Monitoring Tutorial: Getting Started Sending Prometheus Metrics

This Logz.io Infrastructure Monitoring tutorial will cover how to get started with our latest product, our new Prometheus-as-a-Service metrics solution. Engineers monitor metrics to understand CPU and memory utilization for infrastructure, duration and serverless execution, or for network traffic. For more advanced Prometheus metrics monitoring operations, teams can send custom metrics to monitor signals like the number of active users.

Logz.io’s flagship product is Log Management, which delivers a fully-managed ELK Stack. Our customers use this product to monitor and investigate their logs, which tells them what’s going on with their application or services.

But consider how many logs your system generates when it’s on fire or cracking at the edges. How do you identify specific problems? 

Where Metrics Fit In

What are metrics, exactly? Metrics are quantifiable data points around your application (application monitoring) or infrastructure (infrastructure monitoring). They will signal when and where problems have taken or are taking place. From there, you need your logs to diagnose the issue.

For example, a spike in CPU usage (which metrics would indicate) is not the problem itself, rather, it’s a side effect of the problem. Users need to investigate their logs to actually identify the root cause of the issue.

Put simply, metrics are important to monitor the health of your system without waiting for end users to flood your support system.

This infrastructure monitoring tutorial will cover the basics of setup for your Logz.io account and connecting your local Prometheus into Logz.io Infrastructure Monitoring for central managed time series storage and analysis. A follow-up tutorial on building visualizations will be published this week.

Activating your Metrics service account

Log into your Logz.io account. If you don’t already have a Logz.io account, you can create a free trial account by clicking the “Getting Started” button on Logz.io homepage and filling in the form.

On your Logz.io account, click here to get to your Metrics account plan, or click the cogwheel ⚙️ → Settings → Manage accounts, and scroll down to the Metrics account plan section. 

Update the metrics account plan and service details as needed, and hit the button at the bottom right side. Note that the free trial plan grants up to 10,000 unique metrics with a 30-day retention period. 

The account will be created on the fly, and you will see the account activated with the requested plan.

Note that under your Prometheus metrics service plan you can find your metrics account token (under ‘Token’). You’ll use that when configuring Prometheus (or other metrics sources) to send metrics to your account.

Hooray, you’re all set to ingest metrics!

Let’s see how to send your Prometheus metrics to your new account.

Set Up Prometheus & Logz.io

So you’ve got your Prometheus running locally and scraping metrics from your environment? Great. All you need to do is add a few configuration lines to have it remote-write them to your Logz.io account. This “Send Your Data” guide will walk you through it step by step. 

Let’s look at the gist of how you edit your prometheus.yml file in just three steps:

1. Configure ‘global’

1. On the `global` section, add the label p8s_logzio_name and give it a name representing your environment. If you use multiple Prometheus instances for different environments or subsystems, and would like to send metrics from all the instances to your Logz.io Infrastructure Monitoring for centralized monitoring, then this label will help you identify them in your Logz.io metrics account. 

 global:
      external_labels:
        p8s_logzio_name: <environment-label>

2. Remote_Write

Next, add a ‘remote_write’ section (if not already there) and configure it to Logz.io.

remote_write:
      - url: https://<<LISTENER-HOST-URL>>:8053
        bearer_token: <<PROMETHEUS-METRICS-SHIPPING-TOKEN>> 
        remote_timeout: 30s
        queue_config:
          batch_send_deadline: 5s  #default = 5s
          max_shards: 10  #default = 1000
          min_shards: 1
          max_samples_per_send: 500 #default = 100
          capacity: 10000  #default = 500

Then paste your metrics service account token in the placeholder of bearer_token field. You can find the metrics account token on the Manage Accounts page as we’ve seen above. The token and the listener values can also be found in the “Send Your Data” guide:

You may also tune the remote-write according to the remote write load you expect and your desired parallelism levels. 

Now that your Prometheus metrics are sent to your Logz.io Infrastructure Monitoring account and persisted there, you can also reduce the retention configuration on your local instance to save storage space. This is typically done with Prometheus’ --storage.tsdb.retention.time flag (defaults to 15d). Limiting by size (currently experimental) is done with --storage.tsdb.retention.size flag. You can find more information on Prometheus storage settings here.

See the “Send Your Data” guide for more details on configuration.

3. Restart Prometheus

Restart your Prometheus for the configuration to take effect and your Prometheus metrics to be ship to the Logz.io Infrastructure Monitoring service.

Give it a few minutes before heading to the dashboard, where you’ll start to see data appear and ready to use. Simply go to the Explore view and type in a PromQL query on your metrics.

Alternatively, just enter a general query such as prometheus_remote_storage_samples_total to see the remote write samples. If you’re sending metrics from multiple Prometheus instances, you can query on the p8s_logzio_name label with the specific value we configured above for this Prometheus instance:

prometheus_remote_storage_samples_total{p8s_logzio_name="<environment-label>"}

What’s Next?

Once you’ve started sending metrics data to Logz.io, you can follow the rest of this Infrastructure Monitoring tutorial series. Check out how to import and build dashboards for visualization and other advanced analytics features to help with your operations processes.