Shipping Metrics from Hashicorp Consul with ELK and Logz.io 

Microservices interact in so many ways. Load balancers, security authentication, and service discovery are just the tip of the iceberg. It can get confusing, if not outright messy. But why be messy when you can be meshy? This is where service meshes come into play, linking the roles these tools have in a common ‘net’ that ties and weaves the whole architecture together. Hashicorp has produced one of the most popular of these organizational assets — Consul Connect.

Consul is a service mesh that provides a full-featured control plane with service discovery, configuration, and segmentation functionality. Consul uses agents running inside each node of a daemonset to manage microservice connections. This management is known as a control plane, and for Consul it is a pluggable architecture that gives it some flexibility other service meshes like competitors don’t have.

Consul is critical when these microservice interconnections sprawl out with highly scaled apps. The more complex the architecture, the more important the mesh becomes. Consul streamlines load balancing, authentication, authorization, network routing, and interactions between nodes themselves.

Proxies and Service Meshes

Service meshes are split between the aforementƒioned control plane and the data plane. The data plane is subservient to the control plane, and the data plane itself contains many proxies that control comms between microservices.

Consul Connect’s big advantage is its pluggability. In turn, it can deploy a variety of proxies. This contrasts with other meshes that exclusively use one proxy, as other meshes are bound to the popular proxy Envoy (which Consul can also support). Different kinds of proxies include per-service sidecar proxies, connect-aware proxies, and L4s.

Installing Consul

Consul gives you a few installation options. To download from Consul, find the appropriate binary from their list of downloads. They have a choice of Mac, Linux, and Windows of course, but also FreeBSD and Solaris.

On MacOS, you can also download from Homebrew with the command brew install consul.

Our Consul tutorial will cover two different methods. You can also compile it straight from the source on GitHub. Finally, they have a Kubernetes-based installation.

Compile Consul Connect from the Source

Golang is a prereq for Consul. Make sure this installation and configuration is up to speed, then also include a GOPATH variable set.

Next, clone the repository from GitHub:

$ mkdir -p $GOPATH/src/github.com/hashicorp 

# sudo might be a necessary command here

$ git clone https://github.com/hashicorp/consul.git

$ cd consul

Bootstrap with this command to compile tools and libraries:

$ make tools

Use the make dev command to build Consul for your local environment:

$ make dev

Finally, check the installation was successful by checking the version:

$ consul -v

Install Consul on Kubernetes

Presuming you have already installed Helm, fetch the Hashicorp repository:

$ helm repo add hashicorp https://helm.releases.hashicorp.com

Next, check that the Consul chart is present.

$ helm search repo hashicorp/consul

NAME                CHART VERSION   APP VERSION DESCRIPTION

hashicorp/consul    0.20.1          1.7.2       Official HashiCorp Consul Chart

Then, for Helm 3, install with the following command:

$ helm install consul hashicorp/consul --set global.name=consul

Alternatively, for Helm 2, install with this command (the only difference being --name):

$ helm install --name consul hashicorp/consul --set global.name=consul

Next, create a config.yaml file for Consul if you ever want to override default settings.

Then, run

$ helm install consul hashicorp/consul -f config.yaml

Shipping Consul Metrics with ELK

Great! We’ve installed Consul and deployed a sample app that uses Consul features for controlling and routing requests to the application’s services. We can now move on to the next step which is monitoring Consul’s operation using the ELK Stack.

Go into your metricbeat.yml file and activate the Consul module by entering the following:

- module: prometheus
  period: 10s
  metricsets: ["collector"]
  hosts: ["localhost:8500"]
  metrics_path: /metrics

#Note: 8500 is the port number for Consul, not Prometheus. We are using the Prometheus module here to ship metrics, but from Consul itself.

To forward your metrics within the ELK Stack, use the following configs in metricbeat.yml:

Output within the ELK Stack

For forwarding metrics to Elasticsearch:

output.elasticsearch:

  hosts: ["localhost:9200"]

For forwarding metrics to Logstash:

output.logstash:

  hosts: ["localhost:5044"]

If you choose Logstash as an intermediary to Elasticsearch, this sample will show you how to configure Logstash:

input {
    beats {
        port => "5044"
    }
filter {
    grok {
      match => { "message" => "%{CONSULLOG}" }
    }
    date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
  geoip {
      source => "clientip"
    }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
  }
}

Shipping Consul Metrics with Logz.io

For Logz.io, you will need Metricbeat 7.6 or higher. Make sure to update before shipping the metrics.

1. Consul Configuration

Now, you’ll need to update your Consul configuration. Go to a Consul server and create a new file called prometheus.json under the Consul config library at /etc/consul.d. Then, add the following telemetry stanza to the prometheus.json:

{
  "telemetry": {
    "disable_hostname": true,
    "prometheus_retention_time": "72h"

  }
}

2. Consul Server

Next, check that this particular Consul server’s metrics are in Prometheus format at this endpoint:

http://127.0.0.1:8500/v1/agent/metrics?format=prometheus

3. Download the Logz.io Certificate

Then, download the Logz.io public certificate for HTTPS shipping:

sudo curl https://raw.githubusercontent.com/logzio/public-certificates/master/TrustExternalCARoot_and_USERTrustRSAAAACA.crt --create-dirs -o /etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt

4. Metricbeat Modules

Afterwards, set up your Metricbeat modules on your Consul servers. Do this by opening metricbeat.yml. Next, copy & paste the following:

metricbeat.modules:
- module: prometheus
  period: 10s
  hosts: ["localhost:8500"]
  metrics_path: /v1/agent/metrics
  query:
    format: prometheus
  processors:
    - add_fields:
        fields:
          module: consul
- module: system
  metricsets:
    - cpu              # CPU usage
    - load             # CPU load averages
    - memory           # Memory usage
    - network          # Network IO
    - diskio
  enabled: true
  period: 10s
  cpu.metrics:  ["percentages","normalized_percentages"]  # The other available option is ticks.
  processors:
    - add_fields:
        fields:
          module: consul

Next, add the following to the same metricbeat.yml file:

# ===== General =====
fields:
  logzio_codec: json
  token: <<SHIPPING-TOKEN>>
fields_under_root: true

Finally, configure metricbeat.yml in the Outputs section to include Logz.io:

# ===== Outputs =====
output.logstash:
  hosts: ["<<LISTENER-HOST>>:5015"]
    ssl.certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']

Restart Metricbeat and then open your Consul dashboard in Logz.io.

Screenshot of Consul metrics in Logz.io

Conclusion

Consul serves an increasingly critical role in the modern DevOps tool kit. Making sure it runs in force is too essential. Logz.io wants to make sure that the entire DevOps stack fits firmly into its repertoire to meet the needs of its users, especially for the ones who are keen to maintain using Open Source tools in their organization.

Consequently, Hashicorp Consul and other Hashicorp offerings are certainly essentials in our integration collection, which complements the DevOps suite with a Logz.io Terraform Provider and a HashiCorp Vault integration. We’re looking forward to adding more to this arsenal.