Quick start: Get application traces into the Elastic Stack

edit

Quick start: Get application traces into the Elastic Stack

edit

This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.

This guide describes how to:

  • Collect Application Performance Monitoring (APM) data
  • Send APM data to the Elastic Stack
  • Explore and visualize the data in real-time

For feedback and questions, please contact us in the discuss forum.

Prerequisites

edit
  • Please read the Fleet limitations.
  • The APM integration is experimental and has a number of known limitations. Please read the list of known limitations.
  • You need Elasticsearch for storing and searching your data, and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud (recommended), or self-manage the Elastic Stack on your own hardware.

    Here’s what you need for each deployment type:

Step 1: Set up Fleet

edit

Use Fleet in Kibana to get APM data into the Elastic Stack.

The first time you use Fleet, you need to set it up:

  1. Log in to Kibana and go to Management > Fleet.

    Fleet in Kibana
  2. In Fleet, click Settings and change the defaults, if necessary. For self-managed installations, set the URLs for Elasticsearch and Kibana, including the http ports, then save your changes.

    Fleet settings
  3. Enable central management. Click the Agents tab and click Create user and enable central management.

    Fleet showing prompt to enable central management

Step 2: Add an Elastic Agent to Fleet

edit

Don’t confuse Elastic Agent with APM agents–they are different components. In a later step, you’ll instrument your code with APM agents and send the data to an APM Server instance that Elastic Agent spins up.

Elastic Agent is a single, unified agent that you can deploy to hosts or containers to collect data and send it to the Elastic Stack. Behind the scenes, Elastic Agent runs the Beats shippers or Elastic Endpoint required for your configuration.

If you plan on enabling Real User Monitoring (RUM), you must run Elastic Agent centrally. If RUM is disabled, you should run Elastic Agent on edge machines.

To send APM data to the Elastic Stack:

  1. On the Agents tab in Fleet, click Add agent, and look at the deployment instructions under Enroll in Fleet.
  2. As instructed, download and extract the Elastic Agent to your host. To do this quickly from the command line, run:

    curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-7.12.1-darwin-x86_64.tar.gz
    tar xzvf elastic-agent-7.12.1-darwin-x86_64.tar.gz

    See the download page for other installation options.

  3. Back in Fleet, under Choose an agent policy, notice that the default policy is selected. The default policy includes a system integration for collecting logs and metrics from the host system. Use the default policy to get started quickly.
  4. Under Enroll and start the Elastic Agent, copy the install command if it’s available for your platform, or make a note of the Kibana URL and enrollment token that Fleet generated.

    Fleet showing agent enrollment page
  5. From the agent directory, run the appropriate command to install, enroll, and start an Elastic Agent. Note that this command installs Elastic Agent files in the locations described in Installation layout.

    You must run this command as the root user because some integrations require root privileges to collect sensitive data.

    sudo ./elastic-agent install -f --kibana-url=<kibana_url> --enrollment-token=<enrollment_token>  

    kibana_url is the Kibana URL where Fleet is running, and enrollment_token is the enrollment token acquired from Fleet.

    Omit -f to run an interactive installation.

    Because Elastic Agent is installed as an auto-starting service, it will restart automatically if the system is rebooted.

  6. In Fleet, click Continue to go to the Agents tab. You should see the newly enrolled agent.

    Fleet showing enrolled agents

If the status hangs at Enrolling, make sure the elastic-agent process is running.

If you run into problems, see Troubleshoot common problems.

Step 3: Add the APM integration

edit

Next, you’ll browse a catalog of integrations, then add an APM integration to the default policy used by your agent. You use policies to manage settings across a group of agents. An agent policy may contain any number of integrations for collecting observability data from the various services running on your host.

  1. In Kibana, go back to Management > Fleet, and click the Integrations tab. Use the search bar to find the Elastic APM integration.

    Fleet showing APM integration
  2. Click the APM integration to see more details about it, then click Add Elastic APM.

    Fleet showing APM integration overview
  3. On the Add APM integration page, select the default policy.

    In this guide, you add integrations to the default policy created by Kibana. After you learn the basics, you can create your own policies and assign them to agents.

    Fleet Add APM integration page
  4. Under Configure integration, click the down arrow next to Collect application traces. Under Host, define the host and port where APM Server will listen. Inspect or change other settings.
  5. When you’re done, save and deploy the changes.

    The newly added APM integration should appear under Integrations in the default policy, along with the default system-1 integration.

    Fleet showing default agent policy with apm-1 datasource

    All Elastic Agents that use this policy will collect APM data from your instrumented services.

Step 4: Install APM agents

edit

APM agents are written in the same language as your service. To monitor a new service, you must install the agent and configure it with a service name, APM Server host, and Secret token.

  • Service name: The APM integration maps an instrumented service’s name–defined in each APM agent’s configuration– to the index that its data is stored in Elasticsearch. Service names are case-insensitive and must be unique. For example, you cannot have a service named Foo and another named foo. Special characters will be removed from service names and replaced with underscores (_).
  • APM Server URL: The host and port that APM Server listens for events on. This should match the host and port defined when setting up the APM integration.
  • Secret token: Authentication method for APM agent and APM Server communication. This should match the secret token defined when setting up the APM integration.

You can edit your APM integration settings if you need to change the APM Server URL or secret token to match your APM agents.

Download the APM agent

Download the agent jar from Maven Central. Do not add the agent as a dependency to your application.

Start your application with the javaagent flag

Add the -javaagent flag and configure the agent with system properties.

  • Set required service name
  • Set custom APM Server URL (default: http://localhost:8200)
  • Set the base package of your application
java -javaagent:/path/to/elastic-apm-agent-<version>.jar \
     -Delastic.apm.service_name=my-application \
     -Delastic.apm.server_urls=http://localhost:8200 \
     -Delastic.apm.secret_token= \
     -Delastic.apm.application_packages=org.example \
     -jar my-application.jar

Learn more in the agent reference

Step 5: View your data

edit

Back in Kibana, under Observability, select APM. You should see application performance monitoring data flowing into the Elastic Stack!

The built-in apm_user role is not compatible with the APM integration as it only provides read access to apm-* indices. For a list of indices users need access to, see APM data streams

APM app with data

What’s next?

edit
  • Now that data is streaming into the Elastic Stack, take your investigation to a deeper level! Use Elastic Observability to unify your logs, metrics, uptime, and application performance data.
  • Want to protect your endpoints from security threats? Try Elastic Security. Adding endpoint protection is just another integration that you add to the agent policy!
  • Are your eyes bleary from staring at a wall of screens? Create alerts and find out about problems while sipping your favorite beverage poolside.
  • Want Elastic to do the heavy lifting? Use machine learning to detect anomalies.
  • Got everything working like you want it? Roll out your agent policies to other hosts by deploying Elastic Agents across your infrastructure!