Shipping Metrics from Hashicorp Consul with ELK and Logz.io
September 17, 2020
Microservices interact in so many ways. Load balancers, security authentication, and service discovery are just the tip of the iceberg. It can get confusing, if not outright messy. But why be messy when you can be meshy? This is where service meshes come into play, linking the roles these tools have in a common ‘net’ that ties and weaves the whole architecture together. Hashicorp has produced one of the most popular of these organizational assets — Consul Connect.
Consul is a service mesh that provides a full-featured control plane with service discovery, configuration, and segmentation functionality. Consul uses agents running inside each node of a daemonset to manage microservice connections. This management is known as a control plane, and for Consul it is a pluggable architecture that gives it some flexibility other service meshes like competitors don’t have.
More on the subject:
Consul is critical when these microservice interconnections sprawl out with highly scaled apps. The more complex the architecture, the more important the mesh becomes. Consul streamlines load balancing, authentication, authorization, network routing, and interactions between nodes themselves.
Proxies and Service Meshes
Service meshes are split between the aforementƒioned control plane and the data plane. The data plane is subservient to the control plane, and the data plane itself contains many proxies that control comms between microservices.
Consul Connect’s big advantage is its pluggability. In turn, it can deploy a variety of proxies. This contrasts with other meshes that exclusively use one proxy, as other meshes are bound to the popular proxy Envoy (which Consul can also support). Different kinds of proxies include per-service sidecar proxies, connect-aware proxies, and L4s.
Installing Consul
Consul gives you a few installation options. To download from Consul, find the appropriate binary from their list of downloads. They have a choice of Mac, Linux, and Windows of course, but also FreeBSD and Solaris.
On MacOS, you can also download from Homebrew with the command brew install consul.
Our Consul tutorial will cover two different methods. You can also compile it straight from the source on GitHub. Finally, they have a Kubernetes-based installation.
Compile Consul Connect from the Source
Golang is a prereq for Consul. Make sure this installation and configuration is up to speed, then also include a GOPATH variable set.
Next, clone the repository from GitHub:
$ mkdir -p $GOPATH/src/github.com/hashicorp # sudo might be a necessary command here $ git clone https://github.com/hashicorp/consul.git $ cd consul
Bootstrap with this command to compile tools and libraries:
$ make tools
Use the make dev command to build Consul for your local environment:
$ make dev
Finally, check the installation was successful by checking the version:
$ consul -v
Install Consul on Kubernetes
Presuming you have already installed Helm, fetch the Hashicorp repository:
$ helm repo add hashicorp https://helm.releases.hashicorp.com
Next, check that the Consul chart is present.
$ helm search repo hashicorp/consul NAME CHART VERSION APP VERSION DESCRIPTION hashicorp/consul 0.20.1 1.7.2 Official HashiCorp Consul Chart
Then, for Helm 3, install with the following command:
$ helm install consul hashicorp/consul --set global.name=consul
Alternatively, for Helm 2, install with this command (the only difference being --name):
$ helm install --name consul hashicorp/consul --set global.name=consul
Next, create a config.yaml file for Consul if you ever want to override default settings.
Then, run
$ helm install consul hashicorp/consul -f config.yaml
Shipping Consul Metrics with ELK
Great! We’ve installed Consul and deployed a sample app that uses Consul features for controlling and routing requests to the application’s services. We can now move on to the next step which is monitoring Consul’s operation using the ELK Stack.
Go into your metricbeat.yml file and activate the Consul module by entering the following:
- module: prometheus period: 10s metricsets: ["collector"] hosts: ["localhost:8500"] metrics_path: /metrics
#Note: 8500 is the port number for Consul, not Prometheus. We are using the Prometheus module here to ship metrics, but from Consul itself.
To forward your metrics within the ELK Stack, use the following configs in metricbeat.yml:
Output within the ELK Stack
For forwarding metrics to Elasticsearch:
output.elasticsearch: hosts: ["localhost:9200"]
For forwarding metrics to Logstash:
output.logstash: hosts: ["localhost:5044"]
If you choose Logstash as an intermediary to Elasticsearch, this sample will show you how to configure Logstash:
input {
beats {
port => "5044"
}
filter {
grok {
match => { "message" => "%{CONSULLOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
}
Shipping Consul Metrics with Logz.io
For Logz.io, you will need Metricbeat 7.6 or higher. Make sure to update before shipping the metrics.
1. Consul Configuration
Now, you’ll need to update your Consul configuration. Go to a Consul server and create a new file called prometheus.json under the Consul config library at /etc/consul.d. Then, add the following telemetry stanza to the prometheus.json:
{
"telemetry": {
"disable_hostname": true,
"prometheus_retention_time": "72h"
}
}
2. Consul Server
Next, check that this particular Consul server’s metrics are in Prometheus format at this endpoint:
http://127.0.0.1:8500/v1/agent/metrics?format=prometheus
3. Download the Logz.io Certificate
Then, download the Logz.io public certificate for HTTPS shipping:
sudo curl https://raw.githubusercontent.com/logzio/public-certificates/master/TrustExternalCARoot_and_USERTrustRSAAAACA.crt --create-dirs -o /etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt
4. Metricbeat Modules
Afterwards, set up your Metricbeat modules on your Consul servers. Do this by opening metricbeat.yml. Next, copy & paste the following:
metricbeat.modules: - module: prometheus period: 10s hosts: ["localhost:8500"] metrics_path: /v1/agent/metrics query: format: prometheus processors: - add_fields: fields: module: consul - module: system metricsets: - cpu # CPU usage - load # CPU load averages - memory # Memory usage - network # Network IO - diskio enabled: true period: 10s cpu.metrics: ["percentages","normalized_percentages"] # The other available option is ticks. processors: - add_fields: fields: module: consul
Next, add the following to the same metricbeat.yml file:
# ===== General ===== fields: logzio_codec: json token: <<SHIPPING-TOKEN>> fields_under_root: true
Finally, configure metricbeat.yml in the Outputs section to include Logz.io:
# ===== Outputs ===== output.logstash: hosts: ["<<LISTENER-HOST>>:5015"] ssl.certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']
Restart Metricbeat and then open your Consul dashboard in Logz.io.
Screenshot of Consul metrics in Logz.io
Conclusion
Consul serves an increasingly critical role in the modern DevOps tool kit. Making sure it runs in force is too essential. Logz.io wants to make sure that the entire DevOps stack fits firmly into its repertoire to meet the needs of its users, especially for the ones who are keen to maintain using Open Source tools in their organization.
Consequently, Hashicorp Consul and other Hashicorp offerings are certainly essentials in our integration collection, which complements the DevOps suite with a Logz.io Terraform Provider and a HashiCorp Vault integration. We’re looking forward to adding more to this arsenal.
You Might Also Like
AI Agents: Your data sidekick (minus the coffee breaks)
