Index lifecycle management
editIndex lifecycle management
editLifecycle policies allow you to automate the lifecycle of your APM indices as they grow and age. A default policy is applied to each APM data stream, but can be customized depending on your business needs.
Default policies
editBefore 8.15, clusters used index lifecycle management (ILM) to provide the default data retention settings for APM data. As of 8.15, new clusters use data stream lifecycle (DSL) to provide default data retention settings, but the customization for lifecycle is still performed using ILM policies, which are now configured to override the default DSL polices.
For existing clusters upgrading to 8.15, the default ILM policies should still work as expected. You can can continue using ILM or switch the default to DSL explicitly using the data stream API.
For new clusters created in 8.15 or later, if you prefer to continue using ILM,
follow the steps in this guide to create a custom ILM policy and add it to the *@custom
component template for each data stream.
Each APM data stream has its own default lifecycle policy including a delete definition and a rollover definition.
The table below describes the delete definition for each APM data stream. The delete phase permanently removes the index after a time threshold is met.
Data stream | Delete after | Notes |
---|---|---|
|
10 days |
Raw trace event data |
|
90 days |
Raw RUM trace event data, used in the UI |
|
10 days |
Error event data |
|
10 days |
Logs event data |
|
90 days |
Custom application specific metrics |
|
90 days |
Common system metrics and language specific metrics (for example, CPU and memory usage) |
|
90 days |
Aggregated transaction metrics powering the APM UI |
|
180 days |
Aggregated transaction metrics powering the APM UI |
|
390 days |
Aggregated transaction metrics powering the APM UI |
|
90 days |
Aggregated transaction metrics powering the APM UI |
|
180 days |
Aggregated transaction metrics powering the APM UI |
|
390 days |
Aggregated transaction metrics powering the APM UI |
|
90 days |
Aggregated transaction metrics powering the APM UI |
|
180 days |
Aggregated transaction metrics powering the APM UI |
|
390 days |
Aggregated transaction metrics powering the APM UI |
|
90 days |
Aggregated transaction metrics powering the APM UI |
|
180 days |
Aggregated transaction metrics powering the APM UI |
|
390 days |
Aggregated transaction metrics powering the APM UI |
Rollover (writing to a new index) prevents a single index from growing too large and optimizes indexing and search performance.
Rollover occurs after either an age or size metric is met.
The default rollover definition for each APM data stream is applied based on cluster.lifecycle.default.rollover
.
The APM data stream lifecycle policies can be viewed in Kibana:
- Navigate to Stack Management → Index Management → Component Templates.
-
Search for
apm
. -
Look for templates with
@lifecycle
suffix.
Default lifecycle policies can change between minor versions. This is not considered a breaking change as index management should continually improve and adapt to new features.
Configure a custom index lifecycle policy
editMappings and settings for data streams can be customized through the creation of *@custom
component templates,
which are referenced by the index templates created by the Elasticsearch apm-data plugin.
The easiest way to configure a custom index lifecycle policy per data stream is to edit this template.
This tutorial explains how to apply a custom index lifecycle policy to the traces-apm
data stream.
Step 1: View data streams
editThe Data Streams view in Kibana shows you data streams, index templates, and lifecycle policies:
- Navigate to Stack Management → Index Management → Data Streams.
-
Search for
traces-apm
to see all data streams associated with APM trace data. -
In this example, I only have one data stream because I’m only using the
default
namespace. You may have more if your setup includes multiple namespaces.
Step 2: Create an index lifecycle policy
edit- Navigate to Stack Management → Index Lifecycle Policies.
- Click Create policy.
Name your new policy; For this tutorial, I’ve chosen custom-traces-apm-policy
.
Customize the policy to your liking, and when you’re done, click Save policy.
Step 3: Apply the index lifecycle policy
editTo apply your new index lifecycle policy to the traces-apm-*
data stream,
edit the <data-stream-name>@custom
component template.
-
Click on the Component Template tab and search for
traces-apm
. -
Select the
traces-apm@custom
template and click Manage → Edit. -
Under Index settings, set the ILM policy name created in the previous step:
{ "lifecycle": { "name": "custom-traces-apm-policy", "prefer_ilm": true } }
-
Continue to Review and ensure your request looks similar to the image below. If it does, click Create component template.
Step 4: Roll over the data stream (optional)
editTo confirm that the data stream is now using the new index template and ILM policy, you can either repeat step one, or navigate to Dev Tools and run the following:
The result should include the following:
{ "data_streams" : [ { ... "template" : "traces-apm-default", "ilm_policy" : "custom-traces-apm-policy", ... } ] }
The name of the custom index template created in step three |
|
The name of the ILM policy applied to the new component template in step two |
New ILM policies only take effect when new indices are created, so you either must wait for a rollover to occur (usually after 30 days or when the index size reaches 50 GB), or force a rollover using the Elasticsearch rollover API:
POST /traces-apm-default/_rollover/
Namespace-level index lifecycle policies
editIt is also possible to create more granular index lifecycle policies that apply to individual namespaces.
This process is similar to the above tutorial, but includes cloning and modify the existing index template to use
a new *@custom
component template.