Product

Using the Elastic APM Java Agent on Kubernetes

By
Eyal Koren

Elasticsearch and the rest of the Elastic Stack are commonly used for log and metric aggregation in various environments, including Kubernetes. In addition, the Elastic Stack is frequently being used for uptime tracking, with Heartbeat, as well as Application Performance Monitoring (APM), with agents supporting common programming languages, including Java.

There are multiple ways to set up the Elastic APM Java agent, each is suitable for a different use case. When moving to Kubernetes, the combination of these with the broad range of deployment options can become overwhelming. This post will review some of the possible approaches, explaining where and when each may be appropriate and, of course, offer a concrete setup that separates tracing from the application code.

Hands-on examples

In order to provide a real feel of what that's like, this post runs through a few easy to follow examples. I used Spring’s PetClinic sample app for all examples. In order for you to be able to try out everything discussed here, I published all related code and images. By the end of this post, you should be able to get the PetClinic application deployed on Kubernetes and traced with Elastic APM:

Elastic APM prerequisites

First things first. In order to trace your application with Elastic APM, you need to have the Elastic Stack set up. Aside from being a component of the stack, Elastic APM uses Elasticsearch for storing data and Kibana for visualizing it. 

Sounds complicated? It’s not! This can be easily achieved within a few button clicks through the Elasticsearch Service on Elastic Cloud. If you don’t have an account already, start a free, 14-day trial to follow through with the examples below. 

Otherwise, if you want to self manage your Elastic Stack, this has recently become incredibly easy to accomplish in Kubernetes thanks to Elastic Cloud on Kubernetes (ECK) and the release of the Elastic Stack operator. You can go through the quickstart guide, or just deploy ECK and reuse my all-in-one deployment sample, which creates a three-node Elasticsearch cluster, Kibana and Elastic APM server. The benefit of using the all-in-one configuration is that it is consistent with the rest of the examples throughout this post.

Notes for configuring Elastic Cloud on Kubernetes

Elasticsearch cluster creation may take a few minutes. Make sure your k8s cluster has enough resources and be patient before proceeding. See Monitor cluster health and creation progress for more on monitoring Elasticsearch cluster health and creation progress.

Follow through step 3 of the Kibana deployment instructions in order to expose and access Kibana in your browser. 

If you used the all-in-one deployment, the Kibana service is called kibana-kb-http and the elastic user’s password is stored in the elasticsearch-es-elastic-user secret. You can verify both through kubectl get services and kubectl get secrets.

By default the operator manages a private CA and generates a self-signed certificate used to secure the communication between APM agents and the APM server. You can either use your own certificate or obtain the default in order to enable communication from the agents. See more on that in our documentation. For simplicity, I disabled TLS at the APM server by adding: 

tls:
  selfSignedCertificate:
    disabled: true

Reference application deployment

Now that the Elastic Stack is configured, we can start deploying our application. I created a simple PetClinic image, with mostly default configs, based on this Dockerfile and deployed it for sanity check on my k8s cluster using this deployment configuration: 

apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: petclinic 
  namespace: default 
  labels: 
    app: petclinic 
    service: petclinic 
spec: 
  replicas: 1 
  selector: 
    matchLabels: 
      app: petclinic 
  template: 
    metadata: 
      labels: 
        app: petclinic 
        service: petclinic 
    spec: 
      dnsPolicy: ClusterFirstWithHostNet 
      containers: 
      - name: petclinic 
        image: eyalkoren/pet-clinic:without-agent

And exposed it through this service configuration: 

apiVersion: v1 
kind: Service 
metadata: 
  name: petclinic 
  namespace: default 
  labels: 
    app: petclinic 
spec: 
  type: NodePort 
  ports: 
  - protocol: TCP 
    port: 8080 
    targetPort: 8080 
    nodePort: 30001 
  selector: 
    service: petclinic

The eyalkoren/pet-clinic:without-agent image referred above is publicly available in my dockerhub repository, so you can just apply it as is and try it out. Just don’t forget to forward port 8080 for the petclinic service.

Now let’s go over a couple of options for instrumenting the application.

Self-attaching the agent

In some cases, you may want to install the agent by programmatically attaching it during application startup. For example, when modifying the Java command line options or changing the build process of the docker image are not valid options. In such cases, you can add the apm-agent-attach artifact as a regular maven/gradle dependency to your application, call  ElasticApmAttacher.attach() in your application startup code, rebuild the application, rebuild the image and redeploy. However, keep on reading, as there is an alternative approach suggested below that you may find superior.

Installing through the JVM command line

This is currently the most common installation method. It requires having the agent available in the file system on startup and adding the -javaagent argument to the command line.

Modifying the application image

The most obvious way to get it done is to adjust your application image, as demonstrated in this Dockerfile. The deployment configuration would be the same, apart from the environment variables required for the agent setup: 

apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: petclinic 
  namespace: default 
  labels: 
    app: petclinic 
    service: petclinic 
spec: 
  replicas: 1 
  selector: 
    matchLabels: 
      app: petclinic 
  template: 
    metadata: 
      labels: 
        app: petclinic 
        service: petclinic 
    spec: 
      dnsPolicy: ClusterFirstWithHostNet 
      containers: 
      - name: petclinic
    ############### Image with agent and agent config ##################  
        image: eyalkoren/pet-clinic:with-agent 
        env: 
        - name: ELASTIC_APM_SERVER_URL 
          value: "http://apm-server-apm-http:8200" 
        - name: ELASTIC_APM_SERVICE_NAME 
          value: "petclinic" 
        - name: ELASTIC_APM_APPLICATION_PACKAGES 
          value: "org.springframework.samples.petclinic" 
        - name: ELASTIC_APM_ENVIRONMENT 
          value: test 
        - name: ELASTIC_APM_LOG_LEVEL 
          value: DEBUG 
        - name: ELASTIC_APM_SECRET_TOKEN 
          valueFrom: 
            secretKeyRef: 
              name: apm-server-apm-token 
              key: secret-token
    ####################################################################

Note: The value for the ELASTIC_APM_SECRET_TOKEN environment variable is automatically extracted from the apm-server-apm-token secret, which was created during the APM server setup, so there is no concern for a breach. 

As with the example above, the eyalkoren/pet-clinic:with-agent image is publicly available, so all you need to do is apply this configuration, play with the PetClinic app a bit and see your traced data in the APM tab in your Kibana instance.

Install without modifying the application image

Although the method suggested above does the trick, we prefer finding ways that do not require changing our application images or code. Conceptually, separating the application from its monitoring is always a good idea, if possible. 

This can be achieved through the usage of an init container, as long as we have a way to make the Java command line pick up our -javaagent configuration. In this example, I used the standard JVMTI JAVA_TOOL_OPTIONS environment variable, which does not have to be explicitly specified, but rather is picked up automatically by the JVM, if set, when it starts. If your JVM does not support this option, use any other environment variable to do the same — either one already defined in your startup script (like JAVA_OPTS in some servlet container scripts) or add a dedicated empty one that will have no effect, unless being set. 

Our init container relies on an official Elastic APM Java Agent Docker image containing the latest agent version: docker.elastic.co/observability/apm-agent-java:1.12.0. This is the first version we made publicly available, and we intend to keep publishing them on future agent releases. In case you want to build your own Docker image, see this Dockerfile for reference, where v=LATEST can be replaced with any other version.

Lastly, we need to make sure to configure a shared volume for the pod and copy the agent into it from the init container before our application gets started, and we are ready to go: 

apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: petclinic 
  namespace: default 
  labels: 
    app: petclinic 
    service: petclinic 
spec: 
  replicas: 1 
  selector: 
    matchLabels: 
      app: petclinic 
  template: 
    metadata: 
      labels: 
        app: petclinic 
        service: petclinic 
    spec: 
      dnsPolicy: ClusterFirstWithHostNet 
      ###################### Shared volume and init container ##########################
      volumes: 
      - name: elastic-apm-agent 
        emptyDir: {} 
      initContainers: 
      - name: elastic-java-agent 
        image: docker.elastic.co/observability/apm-agent-java:1.12.0 
        volumeMounts: 
        - mountPath: /elastic/apm/agent 
          name: elastic-apm-agent 
        command: ['cp', '-v', '/usr/agent/elastic-apm-agent.jar', '/elastic/apm/agent']
      ##################################################################################      
      containers: 
      - name: petclinic 
        image: eyalkoren/pet-clinic:without-agent
      ######################### Volume path and agent config ###########################
        volumeMounts: 
        - mountPath: /elastic/apm/agent 
          name: elastic-apm-agent 
        env: 
        - name: ELASTIC_APM_SERVER_URL 
          value: "http://apm-server-apm-http:8200" 
        - name: ELASTIC_APM_SERVICE_NAME 
          value: "petclinic" 
        - name: ELASTIC_APM_APPLICATION_PACKAGES 
          value: "org.springframework.samples.petclinic" 
        - name: ELASTIC_APM_ENVIRONMENT 
          value: test 
        - name: ELASTIC_APM_LOG_LEVEL 
          value: DEBUG 
        - name: ELASTIC_APM_SECRET_TOKEN 
          valueFrom: 
            secretKeyRef: 
              name: apm-server-apm-token 
              key: secret-token 
        - name: JAVA_TOOL_OPTIONS 
          value: -javaagent:/elastic/apm/agent/elastic-apm-agent.jar
      ##################################################################################

As you can see, this deployment config is using the exact same eyalkoren/pet-clinic:without-agent image we used originally. In other words, we managed to monitor our Java application with Elastic APM without having to modify our application code, or even its packaging.

Once this is deployed, you should be able to get all the Elastic APM goodness at hand:

Transaction details

Span details

Performance metrics

Error details

Further enhancements for our Kubernetes offering

We are only at the beginning of our Kubernetes integration path, with lots of enhancements to come. We are looking into ways of making the agent installation more native and seamless, requiring even less configuration than suggested above. 

More importantly, we are constantly working to improve our holistic observability offerings and the Kubernetes ecosystem is no exception. Our goal is to provide a Kubernetes-native solution that will tie your APM traces together with logs and metrics with near-zero effort and we do not intend to rest until we get there, so there’s a lot to look forward to.