Introduction to Python Custom Metrics with Logz.io RemoteWrite SDK

We just announced the creation of a new RemoteWrite SDK to support custom metrics from applications using several different languages. This tutorial will give a quick rundown of how to use the Python SDK.

Using these integrations, Prometheus users can send metrics directly to Logz.io using the RemoteWrite protocol without sending them to Prometheus first. Each SDK, while for a separate language, is each capable of working with frameworks like Thanos, Cortex, and of course M3DB. 

The new RemoteWrite SDK is an open source tool from our team here at Logz.io. We started off looking for a way to make it easier for our customers to send us metrics straight from code, and decided to move toward something that would benefit the entire Prometheus and DevOps communities at large. 

The first SDKs allow shipping of metrics from Golang (Go), Python, and Java. While the JAVA SDK uses a Micrometer registry, the Go and Python editions are native integrations based on OpenTelemetry. We plan to add more languages to the arsenal, starting with .NET and Node.js in the near future.

We’ve attempted  to make the process as straightforward as possible. For the next several weeks, we will produce and post tutorials for each respective SDK, starting with Python. 

Configure Python for Shipping Custom Metrics 

The Python implementation is simple enough, but requires a couple quick downloads to get off the ground. There are several variations of metrics with a number of custom parameters you can export. For the purpose of getting the simplest walkthrough possible, we’re going to start off with a very rudimentary shipment: a single metric.

Our integration with Python custom metrics works with the Snappy C library by Google. Snappy is a compression/decompression tool, meant to give you a literal order of magnitude of added speed to ship data compared to other methods like zlib. Originally written for and in C++, it has distributions for a lot of languages and it is extremely popular for Python.

First, install the Snappy C library. For me, I’m using MacOS, so Homebrew:

brew install snappy

On Windows, you would install with pip:

pip install python_snappy-0.5-cp36-cp36m-win_amd64.whl

Next, install the OpenTelemetry SDK and exporter.

pip install opentelemetry-exporter-prometheus-remote-write

You might need to use pip3 if you’re having trouble, as the exporter isn’t compatible with versions of Python 2:

pip3 install opentelemetry-exporter-prometheus-remote-write

Next, instrument your app:

from opentelemetry import metrics
from opentelemetry.exporter.prometheus_remote_write import (
    PrometheusRemoteWriteMetricsExporter,
)
from opentelemetry.sdk.metrics import MeterProvider

# configure the Logz.io listener endpoint and Prometheus metrics account token
exporter = PrometheusRemoteWriteMetricsExporter(
    endpoint="listener.logz.io:8053",
    headers={
        "Authorization": "Bearer <<Logz.io metrics token>>",
    }
)

push_interval = 15


metrics.set_meter_provider(MeterProvider())
meter = metrics.get_meter(__name__)
metrics.get_meter_provider().start_pipeline(meter, exporter, push_interval)

#You can create several kinds of metrics. Here, we'll create a counter
counter = meter.create_counter(
    name="MyCounter",
    description="Description of MyCounter",
    unit="1",
    value_type=int
)
# add labels
labels = {
    "dimension": "value"
}
counter.add(25, labels)

Individual Custom Metric Options

Including counter, you can create one of six  types of metric instruments with OpenTelemetry: updowncounter, valuerecorder, sumobserver, updownsumobserver, valueobserver.

To define a new instrument, enter a variant of  meter.create_xxxx() function.

For instance, let’s look at valuerecorder. Use the same code snippet above, with the section configuring the counter replaced with the following:

# create a valuerecorder instrument
requests_size = meter.create_valuerecorder(
    name="requests_size",
    description="size of requests",
    unit="1",
    value_type=int,
)
# add labels
labels = {
    "dimension": "value"
}
# provide the first data point
requests_size.record(85, labels)

# create a valuerecorder instrument
requests_size = meter.create_valuerecorder(
    name="requests_size",
    description="size of requests",
    unit="1",
    value_type=int,
)
# add labels
labels = {
    "dimension": "value"
}
# provide the first data point
requests_size.record(85, labels)

Instruments report measurements (i.e., the metric values themselves), meters are a class that create instruments; MeterProvider is the entry point for the API and the meters.

Other instruments include an asynchronous counter, asynchronous gauge, asynchronous updowncounter, and histograms.

Better Metrics

This integration shows Logz.io’s commitment to and enthusiasm for OpenTelemetry, and also accommodates developers working in and on original projects. Direct-from-code metrics – plus other forms of telemetry – will see further support from us with new SDK options in other languages in the very near future.

Subscribe to the Logz.io blog for more updates!