Introduction to Collecting Traces with OpenTelemetry

OpenTelemetry (also abbreviated as OTEL) is an increasingly popular open-source observability platform under the Cloud Native Computing Foundation (CNCF), which is currently the most active project in the CNCF after Kubernetes. It was created to establish a unified and vendor-agnostic way for instrumenting, collecting, and exporting telemetry data for your system and application across traces, logs, and metrics. 

Designed to work across multiple programming languages, platforms, and cloud environments, OpenTelemetry specifically aims to provide a consistent approach for generating and collecting telemetry data to help accelerate troubleshooting and improve applications performance.

OpenTelemetry offers client libraries for the various programming languages for instrumenting applications and emit logs, metrics or traces. These libraries include an API, SDK, and potentially auto-instrumentation agents and other supporting libraries. 

The OpenTelemetry Collector is an application that receives data from the OTEL SDK and other sources and exports this data to various services and destinations. The Collector can receive and export the data over multiple protocols, including the official OpenTelemetry Protocol (OTLP), which uses HTTP or gRPC under the hood.

When using OpenTelemetry, you can, for instance,  send your telemetry data (logs, traces, metrics) from multiple applications to a single dashboard. 

In this article, we will review and highlight the benefits of using OpenTelemetry to accelerate observability, and provide an example of using the OTEL SDK.

A Brief History of OpenTelemetry

OpenTelemetry was born back in 2019 when the OpenTracing and the OpenCensus projects – individual open source tools designed for tracing and metrics telemetry collection, respectively – merged to form this new standard. The overall goal of the project was to standardize related tools (API, SDK, etc.) and formats for gathering, transforming, and transmitting telemetry data, in a vendor-agnostic manner. Check out the Essential Guide to OpenTelemetry to learn more about each component and the various supported programming languages.

OTEL is a project incubated by the Cloud Native Computing Foundation (CNCF) and as of January 2023, measured by its GitHub activity, it is the second most active CNCF-backed project, topped only by Kubernetes.Generally speaking, OpenTelemetry has been gaining significant traction in the software development community as an important element of the most commonly used open source observability toolkit. Let’s dig into that trend.

Trends and Uses of OpenTelemetry

OTEL: The New Standard for Observability

Countless organizations and projects are now implementing OTEL to handle their telemetry data because it quickly enables them to reduce the number of tools they need to collect and communicate telemetry. 

In 2021 AWS launched its AWS Distro for OpenTelemetry, and more recently, Microsoft Azure launched its own distro, dubbed Azure Monitor OpenTelemetry Distro. Many other vendors offer their own distro of OpenTelemetry, including Logz.io as we will see below.

Since OTEL itself is an open-source project, this makes it an ideal instrumentation path to many apps and services, thereby allowing them to remain clearly in the realm of open-source software and open specifications.

Applying the OTEL Standard to Collect Traces with Logz.io

One of the many commercial observability solutions supporting OTEL as a telemetry collection and shipping mechanism is Logz.io. The Logz.io Open 360TM platform offers full stack observability based on leading open source including OpenSearch, Prometheus and Jaeger.

You can send metrics and traces to Logz.io directly from your application or microservices using the OpenTelemetry client libraries. Logz.io provides a user-friendly integrated platform to help you monitor, troubleshoot, and professionally debug your OpenTelemetry data.

Using Logz.io, all the telemetry data generated by your apps and microservices – coming from numerous data sources – can be unified, correlated and analyzed in a single platform. Logz.io also provides over 300 integrations, including popular services such as AWS, GCP and Kubernetes. 

In the following sections, we’ll walk through an example of instrumenting an application that can send telemetry data to Logz.io using OTEL, and explain how to set up Logz.io to further explore this data. 

Employing OpenTelemetry in a Web App: A Use Case

The following provides a simple example of a backend web application that will save telemetry data using the OTEL SDK. More precisely, we will be saving traces with the Go SDK for OTEL.

For this use case, we’ll use a backend REST API server in Golang. The data will be saved to a local file, which can then be read and sent to a third-party app, if so desired.

Setting Up the OTEL SDK for use with Logz.io

To get started with Logz.io, you can set up a Free Trial account here.

Getting started on setting up the OTEL data collection, first, we have to employ two constructor functions. The first one, newExporter creates a trace.SpanExporter object with a useful visualization . Then, the newResource function creates a resource.Resource object with the name Logz.io demo, so that we can identify it. 

As a reminder, in this context, a resource is an entity that produces the telemetry data and an exporter allows telemetry data to be emitted:

// Creating the Exporter. Exporter allows telemetry data to be emitted (console or remote system).
func newExporter(w io.Writer) (trace.SpanExporter, error) {
  return stdouttrace.New(
     stdouttrace.WithWriter(w),
     // With human-readable output
     stdouttrace.WithPrettyPrint(),
  )
}


// In the context of telemetry data, a resource represents the entity producing the data.
func newResource() *resource.Resource {
  r, _ := resource.Merge(
     resource.Default(),
     resource.NewWithAttributes(
        semconv.SchemaURL,
        semconv.ServiceNameKey.String("Logz.io demo"),
        semconv.ServiceVersionKey.String("v1.0.0"),
        attribute.String("environment", "demo"),
     ),
  )
  return r
}

Next comes the work to invoke the main function of OTEL. The first thing we need to do here is to set up the SDK and instantiate the required objects for our needs. 

There’s a lot going on here, so let’s dig in. First, we instantiate a new logger to Stdout. Next, we create the local file traces.txt; this is the file that we will write the logs to. That file is then used as the conduit for the exporter, which means that the telemetry data will be saved there.

We then instantiate a resource and, finally, bind the exporter to the resource using the NewTracerProvider function. Now the data saved in the traces.txt file with the exporter will be bound to the newly created resource:

func main() {
  // …
  logger := log.New(os.Stdout, "", 0)

  // the logger will write to the traces.txt file
  file, err := os.Create("traces.txt")
  if err != nil {
    logger.Fatal(err)
  }
  defer file.Close()

  // we instantiate the exporter and pass it the file. The exporter exports the       telemetry data
  exporter, err := newExporter(file)
  if err != nil {
    logger.Fatal(err)
  }

  resourceEntity := newResource()
  // we bind the exporter with our new resource, which produce the telemetry data
  tracerProvider := trace.NewTracerProvider(
    trace.WithBatcher(exporter),
    trace.WithResource(resourceEntity),
  )

  otel.SetTracerProvider(tracerProvider)

   //..
}

Generating Telemetry Data

Now it’s time to look at how we can generate the trace data using OTEL, given the setup from the previous section:

func main() {
  
// ...  ctx, span := otel.Tracer(tracerName).Start(ctx, "Run")
  defer span.End()

  // .. here the server is running and handles an error (username does not exist) when it receives a request to the user endpoint
  if err != nil {
    span.RecordError(err)
    span.SetStatus(codes.Error, err.Error())
  }
}

In this simple code, the server first writes to the file that it’s running (“Run”) and then writes the error to the log file. Here, the error is that the involved username does not exist. 

In the traces.txt file, it’s interesting to see the standard format of OTEL in action. The format is JSON, and the error is logged under the Status object and the Events object. The output looks like this:

{
 "Name": "Run",
 "SpanContext": {
   "TraceID": "384719368474cf130bdd39cffbe0781f",
   "SpanID": "eb123f7615b18f36",
   "TraceFlags": "01",
   "TraceState": "",
   "Remote": false
 },
 "Parent": {
   "TraceID": "00000000000000000000000000000000",
   "SpanID": "0000000000000000",
   "TraceFlags": "00",
   "TraceState": "",
   "Remote": false
 },
 "SpanKind": 1,
 "StartTime": "2023-01-28T20:57:57.676303622-05:00",
 "EndTime": "2023-01-28T20:57:57.676314482-05:00",
 "Attributes": null,
 "Events": [
   {
     "Name": "exception",
     "Attributes": [
       {
         "Key": "exception.type",
         "Value": {
           "Type": "STRING",
           "Value": "*errors.errorString"
         }
       },
       {
         "Key": "exception.message",
         "Value": {
           "Type": "STRING",
           "Value": "username does not exist"
         }
       }
     ],
     "DroppedAttributeCount": 0,
     "Time": "2023-01-28T20:57:57.676313712-05:00"
   }
 ],
 "Links": null,
 "Status": {
   "Code": "Error",
   "Description": "username does not exist"
 },
 "DroppedAttributes": 0,
 "DroppedEvents": 0,
 "DroppedLinks": 0,
 "ChildSpanCount": 0,
 "Resource": [
   {
     "Key": "environment",
     "Value": {
       "Type": "STRING",
       "Value": "demo"
     }
   },
   {
     "Key": "service.name",
     "Value": {
       "Type": "STRING",
       "Value": "Logz.io demo"
     }
   },
   {
     "Key": "service.version",
     "Value": {
       "Type": "STRING",
       "Value": "v1.0.0"
     }
   },
   {
     "Key": "telemetry.sdk.language",
     "Value": {
       "Type": "STRING",
       "Value": "go"
     }
   },
   {
     "Key": "telemetry.sdk.name",
     "Value": {
       "Type": "STRING",
       "Value": "opentelemetry"
     }
   },
   {
     "Key": "telemetry.sdk.version",
     "Value": {
       "Type": "STRING",
       "Value": "1.11.2"
     }
   }
 ],
 "InstrumentationLibrary": {
   "Name": "Logz.io demo",
   "Version": "",
   "SchemaURL": ""
 }
} 

Framing the Benefits of OpenTelemetry

Using OpenTelemetry clearly creates numerous benefits for software development teams with the larger impact of increasing their productivity. We’ll discuss some of the most important strengths of OTEL below.

Automation

Using the OpenTelemetry standard will help automate many tasks and easily integrate with third-party services and other software libraries.

Generating, organizing, and analyzing data can be difficult and time-consuming. OTEL occupies less of your precious time and requires far less software development. 

By using the OTEL SDK, API, among other development tools, you can greatly reduce the amount of code that you need to write to achieve a properly functioning system – while providing  the detailed visibility that you need. Less development time means that you can focus on other important tasks and let OpenTelemetry handle the hard work occurring under the hood.

Less Prone to Human Error

Chances are, handling telemetry data isn’t a core business activity for your organization. When using OTEL, you employ open-source tools including the  SDK and API in your app to become OTEL-compliant. This will typically result in fewer errors, as most of the work is done by the tools themselves.

A Standard Format

Another important benefit is that the telemetry data generated using OpenTelemetry employs a common standard that multiple third-party apps and services also support. This is one of the real beauties of OTEL: Standardized data brings huge flexibility when it comes time to export and analyze the data. 

Logz.io can be  used in turn to analyze this data, providing a platform that makes monitoring and troubleshooting your apps and microservices far easier. In the next section, we’ll provide a quick tutorial on using Logz.io to analyze telemetry data from the app that we just built.

Forwarding and Analyzing Telemetry Data with Logz.io

Let’s first review how to set up Logz.io to explore OTEL telemetry data generated by our app  referenced in the previous section.

Setting up Logz.io

First, navigate to Logz.io and hit the “Free Trial” button:

Next, sign up for the service using Google, your email address, or other available options. Since we will be analyzing traces, hit the “Traces” button on the left and click “Get started”:

In our case, we will be using the Go instrumentation option with localhost, but feel free to use the setting that best fits your own needs.

In the next screen, we select the Local Host option. Again, feel free to select another option; you can visit the Logz.io official documentation site for guidance on this process.

Installing, Configuring, and Running the Collector

First, we need to download and configure the OpenTelemetry collector. From this link, download the correct binary for the system that you want to run the code on. Once you’ve downloaded the collector, rename it otelcol-contrib. 

Next, create a configuration file, config.yaml,in the same directory as otelcol-contrib, and copy the following configuration:

receivers:
 jaeger:
   protocols:
     thrift_compact:
       endpoint: "0.0.0.0:6831"
     thrift_binary:
       endpoint: "0.0.0.0:6832"
     grpc:
       endpoint: "0.0.0.0:14250"
     thrift_http:
       endpoint: "0.0.0.0:14268"
 opencensus:
   endpoint: "0.0.0.0:55678"
 otlp:
   protocols:
     grpc:
       endpoint: "0.0.0.0:4317"
     http:
       endpoint: "0.0.0.0:4318"
 zipkin:
   endpoint: "0.0.0.0:9411"


exporters:
 logzio/traces:
   account_token: "<YOUR_ACCOUNT_TOKEN>"
   region: "us"

 logging:

processors:
 batch:
 tail_sampling:
   policies:
     [
       {
         name: policy-errors,
         type: status_code,
         status_code: {status_codes: [ERROR]}
       },
       {
         name: policy-slow,
         type: latency,
         latency: {threshold_ms: 1000}
       },
       {
         name: policy-random-ok,
         type: probabilistic,
         probabilistic: {sampling_percentage: 10}
       }
     ]

extensions:
 pprof:
   endpoint: :1777
 zpages:
   endpoint: :55679
 health_check:

service:
 extensions: [health_check, pprof, zpages]
 pipelines:
   traces:
     receivers: [opencensus, jaeger, zipkin, otlp]
     processors: [tail_sampling, batch]
     exporters: [logging, logzio/traces]

Now put the config.yaml file in the same directory as the collector’s binary. Don’t forget to change the value of <YOUR_ACCOUNT_TOKEN> to your actual account token.

Finally, we can start the collector with this command in a bash terminal:

./otelcol-contrib --config ./config.yaml

The collector should be running now, and you should see in the something along the lines of “Everything is ready” in the stdout. Begin running and processing data.”

Note that while we showed here the way to work with the vanilla OTEL Collector open source project, Logz.io offers an easier path with its OTEL-based Logz.io Telemetry Collector, which simplifies the YAML configuration as well as the installation of OTEL Collector. Check out the technical documentation for more details.

Sending Traces to Logz.io

With some minor changes to our code from the use case section above, we can now send the traces to Logz.io instead of saving them in a local file. Among other things, we will need to change the exporter so that it passes the data to the collector.

The code looks like the following. Note that the REST API part is very simplified:

package main

import (
  "context"
  "errors"
  "net/http"

  "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
  "go.opentelemetry.io/otel"
  "go.opentelemetry.io/otel/codes"
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace"
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
  "go.opentelemetry.io/otel/propagation"
  "go.opentelemetry.io/otel/sdk/resource"

  sdktrace "go.opentelemetry.io/otel/sdk/trace"
  semconv "go.opentelemetry.io/otel/semconv/v1.12.0"

  "go.opentelemetry.io/otel/trace"
)

// Creating the Exporter. Exporter allows telemetry data to be emitted (console or remote system).
func newHTTPExporter(ctx context.Context) (*otlptrace.Exporter, error) {
  return otlptracehttp.New(ctx,
     otlptracehttp.WithInsecure(),
     otlptracehttp.WithEndpoint("localhost:4318"),
  )
}

// In the context of telemetry data, a resource represents the entity producing the data.
func newResource(ctx context.Context) (*resource.Resource, error) {
  return resource.New(ctx,
     resource.WithAttributes(
        semconv.ServiceNameKey.String("OTEL-Demo-LogzIO"),
     ),
  )
}

So, we see again the exporter constructor, which now sets the endpoint to 4318, the same port as in the config.yaml file for the collector. Next, we simplified the resource constructor and set ServiceNameKey to OTEL-Demo-LogzIO, which is how we’ll soon identify the trace service in Logz.io.

To simplify things, the HTTP server was directly placed in the main function:

func main() {
  ctx := context.Background()

  // we instantiate the HTTP exporter
  exporter, err := newHTTPExporter(ctx)
  if err != nil {
     panic(err)
  }

  // we instantiate the resource
  resourceEntity, err := newResource(ctx)
  if err != nil {
     panic(err)
  }

  // we build a BatchSpanProcessor from the exporter
  bsp := sdktrace.NewBatchSpanProcessor(exporter)

  // we bind the exporter with our new resource
  tracerProvider := sdktrace.NewTracerProvider(
     sdktrace.WithSampler(sdktrace.AlwaysSample()),
     sdktrace.WithResource(resourceEntity),
     sdktrace.WithSpanProcessor(bsp),
  )

  otel.SetTracerProvider(tracerProvider)
  otel.SetTextMapPropagator(propagation.TraceContext{})

  userHandler := otelhttp.NewHandler(http.HandlerFunc(userHandlerFunction), "user")

  http.Handle("/user", userHandler)
  err = http.ListenAndServe(":15698", nil)
  if err != nil {
     panic(err)
  }
  err = tracerProvider.Shutdown(ctx)
  if err != nil {
     panic(err)
  }
}

In this section, just like before, we create an exporter and a resource and bind them. After setting the tracerProvider, we use the “otelhttp” library to set the /user endpoint and run the server in localhost at port 15698. When the endpoint is hit, it runs the userHandlerFunction, which looks like this:

func userHandlerFunction(w http.ResponseWriter, req *http.Request) {
  ctx := req.Context()
  span := trace.SpanFromContext(ctx)
  defer span.End()
  span.AddEvent("handling request on endpoints user")

  // …
  err := errors.New("username does not exist")
  span.RecordError(err)
  span.SetStatus(codes.Error, err.Error())
}

Basically, it adds an event and an error after that. Things are simplified to show a “username does not exist” error.

Taking a Closer Look at Tracing

Now, run your application and hit the endpoint with a web browser, Postman, or any tool that you like, and the traces should appear quite quickly in Logz.io. The collector running from the terminal should also log some traces.

To do this, navigate back to Logz.io, hit the Traces button on the left navigation bar, and click on Jaeger:

You will then land on the trace search page:

Select the service that you want to look at (in my case, OTEL-Demo-LogzIO) and search the previous few minutes. After the traces appear, click on any one to show further detail:

In this image, we can see the error “username does not exist” from the code above and additional details. That is how you can unlock the power of Logz.io with OTEL in your application!

Benefits of Logz.io 

Using Logz.io brings many benefits and is the easiest way to achieve quality observability for your apps and microservices.

Multiple Out-of-the-Box Capabilities

Logz.io has multiple features you can use without the hassle of spending much time configuring. These include:

• Telemetry visualization with prebuilt and customizable monitoring dashboards
• Traces, logs, and metrics analytics
• Search and filtering system to quickly explore telemetry data
• User-friendly interface
• Telemetry data comparison tool (e.g., comparing trace details)
• 300+ integrations to receive logs, traces, and metrics

Saves Time and Effort

Using Logz.io will save you considerable time and effort. Much of the work your engineering team would be required to do in order to display and analyze the involved telemetry data is now taken care of by Logz.io. Some Logz.io clients reported the following gains in their organization:

• 3x faster data analysis to accelerate troubleshooting
• 62% reduction of total data and cost associated with noisy data
• 20% total engineering time saved from offloading observability tasks

Open-Source

As mentioned before, the OpenTelemetry project is open-sourced and CNCF-backed. This means it brings all the advantages of open-source software, including:

• Freedom and flexibility
• Shared maintenance cost 
• Leverage of the open-source community
• Better security

Maps the Gaps

Using Logz.io in coordination with OTEL will help you find the gaps in your instrumentation data and observability by combining advanced analytics and user-friendly capabilities, based on open source. In turn, this enables you to measurably lower troubleshooting and debugging time.

Conclusion

In this article, we covered a brief introduction to OTEL, the benefits of using it, and a quick tutorial to get started with it in your backend server.

You can start using OTEL in your software development project today, and leverage Logz.io’s suite of tools to unlock the potential of full observability.