Distributed tracing

A trace is a group of transactions and spans with a common root. Each trace tracks the entirety of a single request. When a trace travels through multiple services, as is common in a microservice architecture, it is known as a distributed trace.

Distributed tracing enables you to analyze performance throughout your microservice architecture by tracing the entirety of a request — from the initial web request on your front-end service all the way to database queries made on your back-end services.

Tracking requests as they propagate through your services provides an end-to-end picture of where your application is spending time, where errors are occurring, and where bottlenecks are forming. Distributed tracing eliminates individual service’s data silos and reveals what’s happening outside of service borders.

For supported technologies, distributed tracing works out-of-the-box, with no additional configuration required.

Distributed tracing works by injecting a custom traceparent HTTP header into outgoing requests. This header includes information, like trace-id, which is used to identify the current trace, and parent-id, which is used to identify the parent of the current span on incoming requests or the current span on an outgoing request.

When a service is working on a request, it checks for the existence of this HTTP header. If it’s missing, the service starts a new trace. If it exists, the service ensures the current action is added as a child of the existing trace, and continues to propagate the trace.

In this example, Elastic’s Ruby agent communicates with Elastic’s Java agent. Both support the traceparent header, and trace data is successfully propagated.

In this example, Elastic’s Ruby agent communicates with OpenTelemetry’s Java agent. Both support the traceparent header, and trace data is successfully propagated.

In this example, the trace meets a piece of middleware that doesn’t propagate the traceparent header. The distributed trace ends and any further communication will result in a new trace.

All Elastic agents now support the official W3C Trace Context specification and traceparent header. See the table below for the minimum required agent version:

The APM UI’s timeline visualization provides a visual deep-dive into each of your application’s traces:

Elastic agents automatically propagate distributed tracing context for supported technologies. If your service communicates over a different, unsupported protocol, you can manually propagate distributed tracing context from a sending service to a receiving service with each agent’s API.

Sending services must add the traceparent header to outgoing requests.

transaction := apm.DefaultTracer().StartTransaction("GET /", "request") 
traceContext := transaction.TraceContext() 

// Send TraceContext to receiving service
traceparent := apmhttp.FormatTraceparentHeader(traceContext) 
tracestate := traceContext.State.String()

import OpenTelemetryApi
import OpenTelemetrySdk

struct BasicSetter: Setter { 
    func set(carrier: inout [String: String], key: String, value: String) {
        carrier[key] = value
    }
}

let span : Span = ... 
let setter = BasicSetter()
let propagator = W3CTraceContextPropagator()
var headers = [String:String]()

propagator.inject(spanContext: span.context, carrier: &headers, setter:setter) 

let request = URLRequest(...)
request.allHTTPHeaderFields = headers
... // make network request
span.end()

// Hook into a callback provided by the RPC framework that is called on outgoing requests
public Response onOutgoingRequest(Request request) throws Exception {
  Span span = ElasticApm.currentSpan() 
          .startSpan("external", "http", null)
          .setName(request.getMethod() + " " + request.getHost());
  try (final Scope scope = transaction.activate()) {
      span.injectTraceHeaders((name, value) -> request.addHeader(name, value)); 
      return request.execute();
  } catch (Exception e) {
      span.captureException(e);
      throw e;
  } finally {
      span.end(); 
  }
}

string outgoingDistributedTracingData =
    (Agent.Tracer.CurrentSpan?.OutgoingDistributedTracingData
        ?? Agent.Tracer.CurrentTransaction?.OutgoingDistributedTracingData)?.SerializeToString();
// Now send `outgoingDistributedTracingData` to the receiving service

agent.startTransaction('my-service-a-transaction'); 
const traceparent = agent.currentTraceparent; 
sendMetadata(`traceparent: ${traceparent}\n`); 

$distDataAsString = ElasticApm::getSerializedCurrentDistributedTracingData(); 

client.begin_transaction('new-transaction')

elasticapm.get_trace_parent_header('new-transaction') 

# Send `trace_parent_str` to another service

ElasticAPM.with_span "Name" do |span| 
  header = span.trace_context.traceparent.to_header 
  # send the TraceContext Header to a receiving service...
end

Receiving services must parse the incoming traceparent header, and start a new transaction or span as a child of the received context.

// Receive incoming TraceContext
traceContext, _ := apmhttp.ParseTraceparentHeader(r.Header.Get("Traceparent")) 
traceContext.State, _ = apmhttp.ParseTracestateHeader(r.Header["Tracestate"]...) 

opts := apm.TransactionOptions{
	TraceContext: traceContext, 
}
transaction := apm.DefaultTracer().StartTransactionOptions("GET /", "request", opts) 

// Hook into a callback provided by the framework that is called on incoming requests
public Response onIncomingRequest(Request request) throws Exception {
    // creates a transaction representing the server-side handling of the request
    Transaction transaction = ElasticApm.startTransactionWithRemoteParent(request::getHeader, request::getHeaders); 
    try (final Scope scope = transaction.activate()) { 
        String name = "a useful name like ClassName#methodName where the request is handled";
        transaction.setName(name); 
        transaction.setType(Transaction.TYPE_REQUEST); 
        return request.handle();
    } catch (Exception e) {
        transaction.captureException(e);
        throw e;
    } finally {
        transaction.end(); 
    }
}

var transaction2 = Agent.Tracer.StartTransaction("Transaction2", "TestTransaction",
     DistributedTracingData.TryDeserializeFromString(serializedDistributedTracingData));

const traceparent = readTraceparentFromUDPPacket() 
agent.startTransaction('my-service-b-transaction', { childOf: traceparent }) 

$receiverTransaction = ElasticApm::beginCurrentTransaction( 
    'GET /data-api',
    'data-layer',
    /* timestamp */ null,
    $distDataAsString 
);

parent = elasticapm.trace_parent_from_headers(headers_dict) 
client.begin_transaction('processors', trace_parent=parent) 

# env being a Rack env
context = ElasticAPM::TraceContext.parse(env: env) 

ElasticAPM.with_transaction("Do things", trace_context: context) do 
  ElasticAPM.with_span("Do nested thing", trace_context: context) do 
  end
end

Some additional setup may be required to correlate requests correctly with the Real User Monitoring (RUM) agent.

See the RUM distributed tracing guide for information on enabling cross-origin requests, setting up server configuration, and working with dynamically-generated HTML.