Jaeger integration

This feature is experimental and may be changed in a future release. It is not yet available on Elastic Cloud. For feature status on Elastic Cloud, see #212.

Elastic APM integrates with Jaeger, an open-source, distributed tracing system. This integration allows users with an existing Jaeger setup to switch from the default Jaeger backend, to ingesting data with Elastic’s APM Server, storing data in Elasticsearch, and visualizing traces in the APM app. Best of all, this can be done without any instrumentation changes in your application code.

Get started

edit

Connecting your preexisting Jaeger setup to Elastic APM is easy! Configure APM Server to receive Jaeger data, set up sampling, and configure your Jaeger Agents or Jaeger Clients to start sending spans to APM Server.

There are important Caveats to understand about this integration.

Configure APM Server

edit

Enable the correct jaeger endpoint in the apm-server.yml configuration file.

In a typical Jaeger deployment, Clients send spans to Agents, who forward them to Collectors. If this matches your architecture, enable the gRPC endpoint by setting apm-server.jaeger.grpc.enabled to true.

Alternatively, if you’ve configured your Clients to send spans directly to Collectors (bypassing Jaeger Agents), enable the HTTP endpoint by setting apm-server.jaeger.http.enabled to true.
Configure the host and port that APM Server listens on.

Based on the endpoint enabled in the previous step, configure the relevant host and port:
- apm-server.jaeger.grpc.host defaults to localhost:14250.
- apm-server.jaeger.http.host defaults to localhost:14268.

Configure Sampling

edit

The gRPC endpoint supports probabilistic sampling, which can be used to reduce the amount of data that your agents collect and send. Probabilistic sampling makes a random sampling decision based on the configured sampling value. For example, a value of .2 means that 20% of traces will be sampled.

APM Server automatically enables the sampling endpoint when grpc.enabled is set to true. There are two different ways to configure the sampling rate of your Jaeger Agents:

Centrally, with APM Agent configuration (default).
Locally, in each Jaeger client.

Central sampling

edit

Central sampling, with APM Agent configuration, requires the Kibana endpoint to be enabled. This allows Jaeger clients to poll APM Server for the sampling rate. To enable the kibana endpoint, set apm-server.kibana.enabled to true, and point apm-server.kibana.host at the Kibana host that APM Server will communicate with.

The default sampling ratio, as well as per-service sampling rates, can then be configured via the Agent configuration page in the APM app.

If a sampling rate smaller than 1.0 (100%) is configured, transactions per minute will be incorrectly calculated in the APM app. This is because Elastic APM data model is fundamentally different from the Jaeger data model. In the Elastic ecosystem, sampling only impacts spans; transactions are always sampled at 100%. The Jaeger data structure, however, does not have the concept of transactions. Because of this, sampling decisions will apply to both transactions and spans.

Local sampling

edit

If you don’t have access to the APM app, you’ll need to change the Jaeger Client’s sampler.type and sampler.param, enabling you to set the sampling configuration locally in each Client. See the official Jaeger sampling documentation for more information.

Configure Jaeger communication

edit

Jaeger Agent communication with APM Server (gRPC)

edit

As of this writing, the Jaeger Agent binary offers the --reporter.grpc.host-port CLI flag, which can be used to set a static list of collectors for the Jaeger Agent to connect to. The host:port set here should correspond with the value set in apm-server.jaeger.grpc.host.

Optional token-based authorization*

A secret token or API key can be used to ensure only authorized Jaeger Agents can send data to the APM Server. Authorization is off by default, but can be enabled by setting a value in apm-server.jaeger.grpc.auth_tag. When enabled, APM Server looks for a Process tag in each incoming event, and uses it to authorize the Jaeger Agent against the configured auth_tag and secret token or API key. Auth tags will be removed from events after being verified.

Here’s an example that sets the auth_tag and secret_token in APM Server:

apm-server.jaeger.grpc.enabled=true
apm-server.jaeger.grpc.auth_tag=authorization
apm-server.secret_token=qwerty1234

To authorize Jaeger Agent communication, use the --agent.tags CLI flag to pass the corresponding Process tag to the APM Server:

--agent.tags "authorization=Bearer qwerty1234"

See the Jaeger CLI flags documentation for more information.

Jaeger Client communication with APM Server (HTTP)

edit

If you’re using an officially supported Jaeger Client library and want to connect directly to APM Server, you need to update the JAEGER_ENDPOINT configuration property. This is the HTTP endpoint the Client will send spans to. The host:port set here should correspond to the value set in apm-server.jaeger.http.host.

See the relevant supported Jaeger library for more information.

Start sending span data

edit

Data sent from Jaeger Agents or Clients to APM Server should now be visible in the APM app!

« Custom index lifecycle management with APM Server Storage Management »