Index lifecycle management

edit

Index lifecycle policies allow you to automate the lifecycle of your APM indices as they grow and age. A default policy is applied to each APM data stream, but can be customized depending on your business needs.

See ILM: Manage the index lifecycle to learn more.

Default policies

edit

The table below describes the default index lifecycle policy applied to each APM data stream. Each policy includes a rollover and delete definition:

  • Rollover: Using rollover indices prevents a single index from growing too large and optimizes indexing and search performance. Rollover, i.e. writing to a new index, occurs after either an age or size metric is met.
  • Delete: The delete phase permanently removes the index after a time threshold is met.
Data stream Rollover after Delete after Notes

traces-apm

30 days / 50 GB

10 days

Raw trace event data

traces-apm.rum

30 days / 50 GB

90 days

Raw RUM trace event data, used in the UI

logs-apm.error

30 days / 50 GB

10 days

Error event data

logs-apm.app

30 days / 50 GB

10 days

Logs event data

metrics-apm.app

30 days / 50 GB

90 days

Custom application specific metrics

metrics-apm.internal

30 days / 50 GB

90 days

Common system metrics and language specific metrics (for example, CPU and memory usage)

metrics-apm.service_destination_1m

7 days / 50GB

90 days

Aggregated transaction metrics powering the APM UI

metrics-apm.service_destination_10m

14 days / 50GB

180 days

Aggregated transaction metrics powering the APM UI

metrics-apm.service_destination_60m

30 days / 50GB

390 days

Aggregated transaction metrics powering the APM UI

metrics-apm.service_summary_1m

7 days / 50GB

90 days

Aggregated transaction metrics powering the APM UI

metrics-apm.service_summary_10m

14 days / 50GB

180 days

Aggregated transaction metrics powering the APM UI

metrics-apm.service_summary_60m

30 days / 50GB

390 days

Aggregated transaction metrics powering the APM UI

metrics-apm.service_transaction_1m

7 days / 50GB

90 days

Aggregated transaction metrics powering the APM UI

metrics-apm.service_transaction_10m

14 days / 50GB

180 days

Aggregated transaction metrics powering the APM UI

metrics-apm.service_transaction_60m

30 days / 50GB

390 days

Aggregated transaction metrics powering the APM UI

metrics-apm.transaction_1m

7 days / 50GB

90 days

Aggregated transaction metrics powering the APM UI

metrics-apm.transaction_10m

14 days / 50GB

180 days

Aggregated transaction metrics powering the APM UI

metrics-apm.transaction_60m

30 days / 50GB

390 days

Aggregated transaction metrics powering the APM UI

The APM index lifecycle policies can be viewed in Kibana. Navigate to Stack Management / Index Lifecycle Management, and search for apm.

Default ILM policies can change between minor versions. This is not considered a breaking change as index management should continually improve and adapt to new features.

Configure a custom index lifecycle policy

edit

When the APM integration is installed, Fleet creates a default *@custom component template for each data stream. The easiest way to configure a custom index lifecycle policy per data stream is to edit this template.

This tutorial explains how to apply a custom index lifecycle policy to the traces-apm data stream.

Step 1: View data streams

edit

The Data Streams view in Kibana shows you the data streams, index templates, and index lifecycle policies associated with a given integration.

  1. Navigate to Stack Management > Index Management > Data Streams.
  2. Search for traces-apm to see all data streams associated with APM trace data.
  3. In this example, I only have one data stream because I’m only using the default namespace. You may have more if your setup includes multiple namespaces.

    Data streams info

Step 2: Create an index lifecycle policy

edit
  1. Navigate to Stack Management > Index Lifecycle Policies.
  2. Click Create policy.

Name your new policy; For this tutorial, I’ve chosen custom-traces-apm-policy. Customize the policy to your liking, and when you’re done, click Save policy.

Step 3: Apply the index lifecycle policy

edit

To apply your new index lifecycle policy to the traces-apm-* data stream, edit the <data-stream-name>@custom component template.

  1. Click on the Component Template tab and search for traces-apm.
  2. Select the traces-apm@custom template and click Manage > Edit.
  3. Under Index settings, set the ILM policy name created in the previous step:

    {
      "lifecycle": {
        "name": "custom-traces-apm-policy"
      }
    }
  4. Continue to Review and ensure your request looks similar to the image below. If it does, click Create component template.

    Create component template

Step 4: Roll over the data stream (optional)

edit

To confirm that the data stream is now using the new index template and ILM policy, you can either repeat step one, or navigate to Dev Tools and run the following:

GET /_data_stream/traces-apm-default 

The name of the data stream we’ve been hacking on appended with your <namespace>

The result should include the following:

{
  "data_streams" : [
    {
      ...
      "template" : "traces-apm-default", 
      "ilm_policy" : "custom-traces-apm-policy", 
      ...
    }
  ]
}

The name of the custom index template created in step three

The name of the ILM policy applied to the new component template in step two

New ILM policies only take effect when new indices are created, so you either must wait for a rollover to occur (usually after 30 days or when the index size reaches 50 GB), or force a rollover using the Elasticsearch rollover API:

POST /traces-apm-default/_rollover/

Namespace-level index lifecycle policies

edit

It is also possible to create more granular index lifecycle policies that apply to individual namespaces. This process is similar to the above tutorial, but includes cloning and modify the existing index template to use a new *@custom component template.