Introducing Logz.io’s New Metrics Integration for HashiCorp Consul with OpenTelemetry

HashiCorp Consul began as an open-source project for service discovery. It has evolved to provide  other valuable functionality like secure service mesh to help secure microservice architectures based on service identity, but also the ability to achieve repeatable application deployment lifecycles via Network Infrastructure Automation and control access to the service mesh via Consul API Gateway. These features are considered the four core pillars of Consul service networking. 

We use Consul today here at Logz.io, as do hundreds of our customers. Increasingly, we see our customers adopting HashiCorp Cloud Platform (HCP) Consul to automate service networking across their multi-cloud environments. 

At Logz.io, we focus on helping customers achieve visibility in their complex distributed environments by consolidating logging, tracing, and metrics using open-source technologies. With our newest integration with HCP Consul, we are excited to announce we are now a Premier Partner. 

Our new integration leveraging Prometheus and OpenTelemetry collects data from Consul. Consul exposes Prometheus metrics when configured to do so. All you need to do is configure it to expose the Prometheus metrics on an endpoint to be scraped:

How to Expose Prometheus Metrics with Consul

1. Navigate to the Consul server and create a new file called prometheus.json under the Consul config library at /etc/consul.d.
2. Add the following telemetry stanza to the prometheus.json:

{
 "telemetry": {
   "disable_hostname": true,
   "prometheus_retention_time": "72h"  
 }
}

Once this is configured, you can verify that the endpoint is exposing Prometheus metrics by connecting to this URL: http://127.0.0.1:8500/v1/agent/metrics?format=prometheus

If you do not have the OpenTelemetry collector on your host, then create a directory on your host and download the OpenTelemetry collector that is relevant to the operating system of your host.

After downloading the collector, create a configuration file config.yaml

receivers:
 prometheus:
   config:
     scrape_configs:
       - job_name: 'consul-metrics'
         scrape_interval: 15s
         metrics_path: "/v1/agent/metrics"
         static_configs:
           - targets: ['localhost:8500']
 hostmetrics:
   collection_interval: 1m
   scrapers:
     cpu: ##All execpt Mac
     disk: ##All execpt Mac
     load:
     filesystem:
     memory:
     network:
     process: ##Linux & Windows
processors:
 resourcedetection/system:
   detectors: ["system"]
   system:
     hostname_sources: ["os"]
exporters:
 prometheusremotewrite:
   endpoint: https://<<LISTENER-HOST>>:8053
   headers:
     Authorization: Bearer <<PROMETHEUS-METRICS-SHIPPING-TOKEN>>
service:
 pipelines:
   metrics:
     receivers: [prometheus,hostmetrics]
     processors: [resourcedetection/system]
     exporters:
       - prometheusremotewrite
 telemetry:
   logs:
     level: "debug"

In the exporter section, you must replace the placeholders to match your account specifics. (They are indicated by the double angle brackets << >>):

• Replace <<PROMETHEUS-METRICS-SHIPPING-TOKEN>> with a token for the Metrics account you want to ship to.
  Here’s how to look up your Metrics token.
• Replace <<LISTENER-HOST>> with the Logz.io Listener URL for your region, configured to use port 8052 for http traffic, or port 8053 for https traffic. For example, listener.logz.io if your account is hosted on AWS US East, or listener-nl.logz.io if hosted on Azure West Europe.

Finally you can run the collector:
<path/to>/otelcol-contrib --config ./config.yaml

Replace <path/to> with the path to the directory where you downloaded the collector. 

Login to your Logz.io account and look on the bottom of the page to send the metrics to auto install the dashboard. Your data should look something like this. You are welcome to customize the dashboard from this point. 

Expect more great content as we continue to support more HashiCorp’s products, our Terraform operator has been very popular with customers to automate the Logz.io platform. 

You can find our partner page on the HashiCorp site, Logz.io | HashiCorp Tech Partner. Also don’t forget to star and download the Github page for Consul.