Instrumentation for C# .NET Apps with OpenTelemetry

OpenTelemetry is the recommended path today for instrumenting applications with tracing in a standard, vendor-agnostic and future-proof way. In fact, OpenTelemetry (nicknamed OTEL) encompasses all three pillars of observability: tracing, metrics, and logs. The tracing element of the specification is now stable with the rest following. This is innovative stuff! You can read more on OpenTelemetry and the current release state on this guide.

This article provides a guide to implementing OpenTelemetry tracing in an ASP.NET Core 3.1 application, using the new OpenTelemetry .NET SDK. It covers the following topics:

• Instrumenting ASP.NET Core applications with OpenTelemetry using automatic and manual instrumentation options
• Exporting traces using the OpenTelemetry Protocol to a local OpenTelemetry Collector instance
• Exporting traces from the collector to a tracing backend (in our case, to Logz.io’s managed Jaeger service)

The Example

The complete example code used here is available on GitHub, if you wish to run it yourself.

In our example, there are two ASP.NET Core Web APIs. There’s Service A, which listens on port 5001 on a /ping endpoint, and Service B, which listens on port 6001 on a /ping endpoint.

When Service A receives a ping, it pings Service B. Both services respond with an HTTP status code 200, assuming we have no unexpected failures.

After instrumentation, Service A will emit a span when invoked, and similarly Service B will emit a span when Service A calls it.

An OpenTelemetry Collector receives spans from both services, which we run ourselves locally. The collector then sends the spans to a Logz.io backend, where the request trace is constructed from the spans and visualized in the UI.

Starting the Example Services

The ASP.NET services and the OpenTelemetry collector run in Docker containers, which we build and run with either start.bat or start.sh. This triggers a Docker build for both ASP.NET services and a docker-compose-up, which runs both services and our OpenTelemetry collector.

Trying out the Example Services

Send a GET request to http://localhost:5001/ping. If you get a 200 back, it worked. Then it’s time to read on and find out how to see the traces in a Jaeger UI. 

Adding Tracing to the .NET Application with OpenTelemetry

Several libraries complement the .NET OpenTelemetry implementation that makes integration straightforward. For instrumenting tracing in ASP.NET Core, we use OpenTelemetry.Instrumentation.AspNetCore.

Step 1: Add All of the Necessary Packages

Several libraries complement the OpenTelemetry .NET SDK that makes integration straightforward. For the example services, we have used the following packages:

• OpenTelemetry.Exporter.Console: To output traces to the console during development. 
• OpenTelemetry.Exporter.OpenTelemetryProtocol: To export our traces to our OpenTelemetry Collector using OpenTelemetry Protocol (OTLP).
• OpenTelemetry.Extensions.Hosting: To register the .NET OpenTelemetry provider.
• OpenTelemetry.Instrumentation.AspNetCore: To collect telemetry about incoming web requests.
• OpenTelemetry.Instrumentation.Http: To collect telemetry about outgoing web requests.

Add in these using your usual method, either through the package manager UI in your IDE or via the command line. For example:

dotnet add package OpenTelemetry.Instrumentation.AspNetCore

Step 2: Configure the Trace Provider

Now we can enable the instrumentation with a single block of code in our startup to:

• Add a trace provider for OpenTelemetry
• Set the service name we want to appear in the trace
• Add the ASP.NET Core instrumentation
• Add an exporter using the OpenTelemetry protocol (OTLP) over gRPC pointing to the OpenTelemetry Collector instance

The code looks like this:

public void ConfigureServices(IServiceCollection services)
{
 
    // ...
    
    services.AddOpenTelemetryTracing(
        (builder) => builder
            .SetResourceBuilder(ResourceBuilder.CreateDefault()
                .AddService("ServiceA"))
            .AddAspNetCoreInstrumentation()
            .AddOtlpExporter(o =>
            {
                o.Endpoint = new Uri("http://otel-collector:4317");
            }));
}

That’s all the coding you need! The libraries we used above provide auto-instrumentation of all the incoming and outgoing web requests.

Note that we’re using port 4317, which is the default port for OTLP/gRPC in the OpenTelemetry specification at the time of writing. Make sure this port is available on your system.

Step 3: (Optional) Verify Correct Instrumentation Using Console Output

If you’re keen to see some trace output straight away, replace AddOtlpExporter(…) with AddConsoleExporter.

You need to include an additional package for this to work:

dotnet add package OpenTelemetry.Exporter.Console

Now, when we send a GET request to http://localhost:5001/ping on our new ASP.NET API for Service A, we get the trace output in the console:

Activity.Id:          |e773ec8c-486851cb3f670e0f.
Activity.ActivitySourceName: OpenTelemetry.Instrumentation.AspNetCore
Activity.DisplayName: Ping
Activity.Kind:        Server
Activity.StartTime:   2021-07-18T09:01:21.0406870Z
Activity.Duration:    00:00:00.3602330
Activity.TagObjects:
    http.host: localhost:5001
    http.method: GET
    http.path: /ping
    http.url: http://localhost:5001/ping
    http.user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)
    http.route: Ping
    http.status_code: 200
    otel.status_code: UNSET
Resource associated with Activity:
    service.name: ServiceA
    service.instance.id: 4a293b2d-3f0b-4898-8b5c-a710ae439471

Step 4: Set Up OpenTelemetry Collector to Collect and Export Traces to our Backend

So that we can focus on exactly how we get these traces from our app to Jaeger, we avoid a context switch to the Jaeger setup and instead send our traces directly to a Jaeger SaaS backend at Logz.io. 

But before we can do that, we need an OpenTelemetry Collector. The collector will take on two roles: 

• Receive the spans from across our services in OTLP format over gRPC
• Then export these spans to the tracing backend of choice

We’ve already configured our apps to export to an OpenTelemetry Collector:

.AddOtlpExporter(o =>
{
o.Endpoint = new Uri("http://otel-collector:4317");
}));

And our complete example on GitHub fires up a Docker container for this collector:

otel-collector:
  image: otel/opentelemetry-collector-contrib:0.23.0
  container_name: otel-logzio
  command: ["--config=/etc/otel-collector-config.yml"]
  volumes:
    - ./config.yaml:/etc/otel-collector-config.yml:ro
  ports:
    - "4317:4317"

Note that we chose to export to Logz.io, but there are many other exporters and receivers available for OpenTelemetry Collector, . 

The collector’s config.yaml file for our example is quite simple as we’re only looking to support one receiver and one exporter:

receivers:
  otlp:
    protocols:
      grpc:
 
exporters:
  logzio:
    account_token: "<<YOUR_TRACING_SHIPPING_TOKEN>>"
    #region: "<<YOUR_LOGZIO_ACCOUNT_REGION_CODE>>" - (Optional)
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logzio]

Replace <<YOUR_TRACING_SHIPPING_TOKEN>> with your Logz.io account token. If you don’t have an account you can get a free one here. If your account isn’t on US East, specify the region code.

Step 5: Run it all together and verify in Jaeger UI

Fire up all the Docker containers with start.bat (or start.sh) again and send a GET request to http://localhost:5001/ping (Service A).

Then, from your Logz.io dashboard, switch to the Tracing tab and search for Service A in the Jaeger UI:

Click the summary to expand the full trace and to see both spans and the time they took:

We can see the full span for the time Service A was processing the GET request. In our Service A controller, we send a GET request to Service B:

using var client = new HttpClient();
_ = await client.GetAsync("http://aspcore-service-b:6001/ping");

Service B also records a trace span for handling that request.

Adding Manual Instrumentation to Your App

Getting all our web requests instrumented was super simple with auto-instrumentation. But there might be lots going on in our services, and it would be helpful if we broke the span down into parts for finer-grain tracing. To do this, we can add additional spans manually over sections of the code. 

Note that OpenTelemetry .NET maintains compatibility with existing .NET tracing, and so a span is an Activity. 

We can modify our startup for Service A to include a new tracer source, ExampleTracer:

public void ConfigureServices(IServiceCollection services) 
{ 
	services.AddOpenTelemetryTracing( 
    	(builder) => builder 
        	.SetResourceBuilder(ResourceBuilder.CreateDefault() 
            	.AddService("ServiceA")) 
            .AddSource("ExampleTracer") 
  
    	// ... 
} 

Then we can generate a new manual span by starting a new Activity, and these spans will be sent to our controller. In the example below, we have a span for the HTTP call to Service B and another with a slight wait for illustrative purposes.

public class PingController : ControllerBase
{
    [HttpGet]
    public async Task<IActionResult> Get()
    {
        using var source = new ActivitySource("ExampleTracer");
 
        // A span
        using var activity = source.StartActivity("Call to Service B");
 
        // 'Ping' Service B
        using var client = new HttpClient();
        _ = await client.GetAsync("http://aspcore-service-b:6001/ping");
 
        // Another span
        using var activityTwo = source.StartActivity("Arbitrary 10ms delay");
        await Task.Delay(10);
 
        return Ok();
    }
}

Start everything up, fire a GET request at Service A, and return to your Jaeger UI at Logz.io. You now see the new spans:

Adding User-Defined Context with Baggage

The OpenTelemetry specification allows for the movement of trace information across service boundaries through a span context. This information, which includes identifiers for the span and overall trace, makes it possible to follow the flow through the system. 

OpenTelemetry also offers a correlation context that corresponds to the baggage property. This carries user-defined properties across service boundaries. In the .NET library, we can set them as follows: 

Baggage.Current.SetBaggage("ExampleItem", "The information"); 

If we add baggage in Service A:

public class PingController : ControllerBase
{
    [HttpGet]
    public async Task<IActionResult> Get()
    {
        using var source = new ActivitySource("ExampleTracer");
 
        // A span
        using var activity = source.StartActivity("Call to Service B");
 
        Baggage.Current.SetBaggage("ExampleItem", "The information");
 
        // 'Ping' Service B
        using var client = new HttpClient();
        _ = await client.GetAsync("http://aspcore-service-b:6001/ping");
 
        // Another span
        using var activityTwo = source.StartActivity("Arbitrary 10ms delay");
        await Task.Delay(10);
 
        return Ok();
    }
}

We are then able to extract this information from the context in Service B and, for example, add it as a tag in the span.

public class PingController : ControllerBase
{
    [HttpGet]
    public IActionResult Get()
    {
        var infoFromContext = Baggage.Current.GetBaggage("ExampleItem");
 
        using var source = new ActivitySource("ExampleTracer");
 
        // A span
        using var activity = source.StartActivity("In Service B GET method");
        activity?.SetTag("InfoServiceBReceived", infoFromContext);
        return Ok();
    }
}

You can see this tag in the span when viewing the trace in the Jaeger UI:

This has worked because we have plugged in a library that instruments HTTP requests to Service A:

.AddHttpClientInstrumentation()

Where other protocols are used to communicate with services downstream (and there are no available instrumentation libraries), then baggage can be injected manually using any one of the W3C compliant propagators.

Conclusion

We’ve covered everything you need to start with OpenTelemetry in ASP.NET. We began by exploring how to instrument OpenTelemetry tracing in an ASP.NET Core application using automatic and manual instrumentation options. Then we discussed how to export those traces to an OpenTelemetry Collector, and from there on to our backend tool of choice for analysis.

If you are interested in trying this integration out using Logz.io backend, feel free to sign up for a free account and then use our documentation to set up instrumentation for your own .NET application.