Tutorial: Customize data retention policies
editTutorial: Customize data retention policies
editThis tutorial explains how to apply a custom ILM policy to an integration’s data stream.
Scenario: You have Elastic Agents collecting system metrics with the System integration in two environments—one with the namespace development
, and one with production
.
Goal: Customize the ILM policy for the system.network
data stream in the production
namespace.
Specifically, apply the built-in 90-days-default
ILM policy so that data is deleted after 90 days.
Step 1: View data streams
editThe Data Streams view in Kibana shows you the data streams, index templates, and ILM policies associated with a given integration.
- Navigate to Stack Management > Index Management > Data Streams.
-
Search for
system
to see all data streams associated with the System integration. -
Select the
metrics-system.network-{namespace}
data stream to view its associated index template and ILM policy. As you can see, the data stream follows the Data stream naming scheme and starts with its type,metrics-
.
Step 2: Create a component template
editFor your changes to continue to be applied in future versions,
you must put all custom index settings into a component template.
The component template must follow the data stream naming scheme,
and end with @custom
:
<type>-<dataset>-<namespace>@custom
For example, to create custom index settings for the system.network
data stream with a namespace of production
,
the component template name would be:
metrics-system.network-production@custom
- Navigate to Stack Management > Index Management > Component Templates
- Click Create component template.
-
Use the template above to set the name—in this case,
metrics-system.network-production@custom
. Click Next. -
Under Index settings, set the ILM policy name under the
lifecycle.name
key:{ "lifecycle": { "name": "90-days-default" } }
-
Continue to Review and ensure your request looks similar to the image below. If it does, click Create component template.
Step 3: Clone and modify the existing index template
editNow that you’ve created a component template, you need to create an index template to apply the changes to the correct data stream. The easiest way to do this is to duplicate and modify the integration’s existing index template.
When duplicating the index template, do not change or remove any managed properties. This may result in problems when upgrading.
- Navigate to Stack Management > Index Management > Index Templates.
-
Find the index template you want to clone. The index template will have the
<type>
and<dataset>
in its name, but not the<namespace>
. In this case, it’smetrics-system.network
. - Select Actions > Clone.
-
Set the name of the new index template to
metrics-system.network-production
. -
Change the index pattern to include a namespace—in this case,
metrics-system.network-production*
. This ensures the previously created component template is only applied to theproduction
namespace. -
Set the priority to
250
. This ensures that the new index template takes precedence over other index templates that match the index pattern. -
Under Component templates, search for and add the component template created in the previous step.
To ensure your namespace-specific settings are applied over other custom settings,
the new template should be added below the existing
@custom
template. - Create the index template.
Step 4: Roll over the data stream (optional)
editTo confirm that the data stream is now using the new index template and ILM policy, you can either repeat step one, or navigate to Dev Tools and run the following:
The result should include the following:
{ "data_streams" : [ { ... "template" : "metrics-system.network-production", "ilm_policy" : "90-days-default", ... } ] }
The name of the custom index template created in step three |
|
The name of the ILM policy applied to the new component template in step two |
New ILM policies only take effect when new indices are created, so you either must wait for a rollover to occur (usually after 30 days or when the index size reaches 50 GB), or force a rollover using the Elasticsearch rollover API:
POST /metrics-system.network-production/_rollover/