OrionIQ Agentic Observability
Distributed tracing is a technique for recording and visualizing the path a single request takes as it moves through a distributed system made up of multiple services, processes, or containers. Each service the request touches generates a span, a record of the work performed, its duration, and its outcome. All the spans generated by one request share a trace ID, and together they form a trace that shows the full journey of that request from entry point to completion.
The concept grew out of a practical problem in distributed and microservices architectures: a single user-facing action can trigger calls across dozens of independent services, and when something goes wrong, no single service has the full picture. Distributed tracing solves this by reconstructing that picture after the fact, connecting fragments of activity from many services into one coherent timeline.
A trace begins the moment a request enters a system, typically at an API gateway or load balancer, where it’s assigned a unique trace ID. As the request is passed to downstream services, that trace ID travels with it, usually embedded in HTTP headers or message queue metadata. Each service that handles part of the request creates one or more spans, recording when its work started, when it finished, and any relevant metadata such as an error code or a database query.
Spans are structured hierarchically. A parent span might represent an entire API call, while child spans represent the individual steps within it, such as a cache lookup, a database query, and a call to a third-party service. When all the spans are collected and reassembled by trace ID, the result is a structured view of the request’s full path, often displayed as a waterfall diagram or a flame graph, where the width of each bar represents time spent and the position shows the sequence of calls.
Context propagation, the mechanism that carries the trace ID and related metadata between services, is what makes this reconstruction possible. Standards like the W3C Trace Context specification define a consistent format for this propagation, so that services written in different languages or maintained by different teams can still participate in the same trace without custom integration work.
A trace represents the complete journey of one request across every service it touches. A span represents a single unit of work within that journey, tied to one specific service or operation. A single trace can contain anywhere from a handful of spans to hundreds, depending on how many services and internal operations were involved in handling the request.
Spans carry their own metadata independent of the trace as a whole, including start and end timestamps, tags describing the operation, and references to their parent span. This structure is what allows tracing systems to show not just that a request was slow, but exactly which span within that request accounted for the delay.
Microservices architectures split functionality across many small, independently deployed services, which improves scalability and development speed but removes the built-in visibility a monolithic application has into its own internal calls. Without tracing, diagnosing a slow or failed request means manually correlating logs and metrics across every service that might have been involved, a process that becomes slower and less reliable as the number of services grows.
Distributed tracing addresses this by giving engineers a single artifact, the trace, that shows the entire request path along with timing for each step. This makes it possible to identify which service introduced latency or failure without first having to guess which services were even involved.
OpenTelemetry is an open standard that defines how tracing, along with metrics and logs, should be instrumented, collected, and exported. Before OpenTelemetry, teams instrumenting for tracing often had to use vendor-specific SDKs, which made it harder to switch tracing backends or support new languages without redoing instrumentation work.
OpenTelemetry standardizes the instrumentation layer, meaning code written to emit OpenTelemetry traces can send that data to any compatible backend. This has made OpenTelemetry the default choice for new tracing implementations, since it decouples the decision of how to instrument code from the decision of where to store and analyze the resulting data.
The purpose of distributed tracing is to reconstruct the full path of a request across multiple services, so engineers can see where time was spent and where failures occurred without manually correlating separate logs and metrics.
A trace ID is a unique identifier assigned to a request when it enters a system. It’s passed along to every service that processes part of that request, allowing all the resulting spans to be grouped together into a single trace.
Context propagation is the process of passing trace identifiers and related metadata between services as a request moves through a system, typically through HTTP headers or message metadata, using standards like W3C Trace Context.
Application logging records discrete events within a single service, while distributed tracing connects related events across multiple services into one timeline tied to a specific request. Logs and traces are often used together, with traces pointing to the relevant service and logs providing additional detail from that service.
A flame graph is a visual representation of a trace, showing spans as horizontal bars where width represents duration and vertical position represents the call hierarchy. It gives engineers a fast way to spot which span in a trace took the most time.
