This is the second part of a two part series. The first part is available at Zero config OpenTelemetry auto-instrumentation for Kubernetes Java applications. In that first part I walk through setting up and installing the OpenTelemetry Operator for Kubernetes, and configuring that for auto-instrumentation of a Java application using the OpenTelemetry Java agent.
In this second part, I show how to install any Java agent via the OpenTelemetry operator, using the Elastic Java agents as examples.
Installation and configuration recap
Part 1 of this series, Zero config OpenTelemetry auto-instrumentation for Kubernetes Java applications, details the installation and configuration of the OpenTelemetry operator and an Instrumentation resource. Here is an outline of the steps as a reminder:
- Install cert-manager, eg kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.4/cert-manager.yaml
- Install the operator, eg kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
- Create an Instrumentation resource
- Add an annotation to either the deployment or the namespace
- Deploy the application as normal
In that first part, steps 3, 4 & 5 were implemented for the OpenTelemetry Java agent. In this blog I’ll implement them for other agents, using the Elastic APM agents as examples. I assume that steps 1 & 2 outlined above have already been done, ie that the operator is now installed. I will continue using the
Using the Elastic Distribution for OpenTelemetry Java
From version 0.4.0, the Elastic Distribution for OpenTelemetry Java includes the agent jar at the path
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: elastic-otel
namespace: banana
spec:
exporter:
endpoint: https://my.apm.server.url
propagators:
- tracecontext
- baggage
- b3
sampler:
type: parentbased_traceidratio
argument: "1.0"
java:
image: docker.elastic.co/observability/elastic-otel-javaagent:0.5.0
env:
- name: OTEL_EXPORTER_OTLP_HEADERS
value: "Authorization=Bearer my-apm-secret-token"
- name: ELASTIC_OTEL_INFERRED_SPANS_ENABLED
value: "true"
- name: ELASTIC_OTEL_SPAN_STACK_TRACE_MIN_DURATION
value: "50"
I’ve included environment for switching on several features in the agent, including
- ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED to switch on the inferred spans implementation feature described in this blog
- Span stack traces are automatically captured if the span takes more than ELASTIC_OTEL_SPAN_STACK_TRACE_MIN_DURATION (default would be 5ms)
Adding in the annotation ...
metadata:
annotations:
instrumentation.opentelemetry.io/inject-java: "elastic-otel"
... to the pod yaml gets the application traced, and displayed in the Elastic APM UI, including the inferred child spans and stack traces
The additions from the features mentioned above are circled in red - inferred spans (for methodC and methodD) bottom left, and the stack trace top right. (Note that the pod included the
Using the Elastic APM Java agent
From version 1.50.0, the Elastic APM Java agent includes the agent jar at the path /javaagent.jar in the docker image - which is essentially all that is needed for a docker image to be usable by the OpenTelemetry operator for auto-instrumentation. This means the Instrumentation resource is straightforward to define:
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: elastic-apm
namespace: banana
spec:
java:
image: docker.elastic.co/observability/apm-agent-java:1.51.0
env:
- name: ELASTIC_APM_SERVER_URL
value: "https://my.apm.server.url"
- name: ELASTIC_APM_SECRET_TOKEN
value: "my-apm-secret-token"
- name: ELASTIC_APM_LOG_LEVEL
value: "INFO"
- name: ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED
value: "true"
- name: ELASTIC_APM_LOG_SENDING
value: "true"
I’ve included environment for switching on several features in the agent, including
- ELASTIC_APM_LOG_LEVEL set to the default value (INFO) which could easily be switched to DEBUG
- ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED to switch on the inferred spans implementation equivalent to the feature described in this blog
- ELASTIC_APM_LOG_SENDING which switches on sending logs to the APM UI, the logs are automatically correlated with transactions (for all common logging frameworks)
Adding in the annotation ...
metadata:
annotations:
instrumentation.opentelemetry.io/inject-java: "elastic-apm"
... to the pod yaml gets the application traced, and displayed in the Elastic APM UI, including the inferred child spans
(Note that the pod included the
Using an extension with the OpenTelemetry Java agent
Setting up an Instrumentation resource for the OpenTelemetry Java agent is straightforward and was done in part 1 of this two part series - and you can see from the above examples it’s just a matter of deciding on the docker image URL you want to use. However if you want to include an extension in your deployment, this is a little more complex, but also supported by the operator. Basically the extensions you want to include with the agent need to be in docker images - or you have to build an image which includes the extensions that are not already in images. Then you declare the images and the directories the extensions are in, in the Instrumentation resource. As an example, I’ll show an Instrumentation which uses version 2.5.0 of the OpenTelemetry Java agent together with the inferred spans extension from the Elastic OpenTelemetry Java distribution. The distro image includes the extension at path
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: otel-plus-extension-instrumentation
namespace: banana
spec:
exporter:
endpoint: https://my.apm.server.url
propagators:
- tracecontext
- baggage
- b3
sampler:
type: parentbased_traceidratio
argument: "1.0"
java:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:2.5.0
extensions:
- image: "docker.elastic.co/observability/elastic-otel-javaagent:0.5.0"
dir: "/extensions"
env:
- name: OTEL_EXPORTER_OTLP_HEADERS
value: "Authorization=Bearer my-apm-secret-token"
- name: ELASTIC_OTEL_INFERRED_SPANS_ENABLED
value: "true"
Note that you can have multiple
Next steps
Here I’ve shown how to use any agent with the OpenTelemetry Operator for Kubernetes, and configure that for your system. In particular the examples have showcased how to use the Elastic Java agents to auto-instrument Java applications running in your Kubernetes clusters, along with how to enable features, using Instrumentation resources. And you can set it up for either zero config for deployments, or for just one annotation which is generally a more flexible mechanism (you can have multiple Instrumentation resource definitions, and the deployment can select the appropriate one for its application).
The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.