Tutorial: Data stream retention

In this tutorial, we are going to go over the data stream lifecycle retention; we will define it, go over how it can be configured and how it can gets applied. Keep in mind, the following options apply only to data streams that are managed by the data stream lifecycle.

You can verify if a data steam is managed by the data stream lifecycle via the get data stream lifecycle API:

resp = client.indices.get_data_lifecycle(
    name="my-data-stream",
)
print(resp)

response = client.indices.get_data_lifecycle(
  name: 'my-data-stream'
)
puts response

const response = await client.indices.getDataLifecycle({
  name: "my-data-stream",
});
console.log(response);

GET _data_stream/my-data-stream/_lifecycle

The result should look like this:

{
  "data_streams": [
    {
      "name": "my-data-stream",                                   
      "lifecycle": {
        "enabled": true                                           
      }
    }
  ]
}

We define retention as the least amount of time the data of a data stream are going to be kept in Elasticsearch. After this time period has passed, Elasticsearch is allowed to remove these data to free up space and/or manage costs.

We define 4 different types of retention:

The effective is calculated in the following way:

The above is demonstrated in the examples below:

Considering our example, if we retrieve the lifecycle of my-data-stream:

resp = client.indices.get_data_lifecycle(
    name="my-data-stream",
)
print(resp)

response = client.indices.get_data_lifecycle(
  name: 'my-data-stream'
)
puts response

const response = await client.indices.getDataLifecycle({
  name: "my-data-stream",
});
console.log(response);

GET _data_stream/my-data-stream/_lifecycle

We see that it will remain the same with what the user configured:

{
  "global_retention" : {
    "max_retention" : "90d",                                   
    "default_retention" : "7d"                                 
  },
  "data_streams": [
    {
      "name": "my-data-stream",
      "lifecycle": {
        "enabled": true,
        "data_retention": "30d",                                
        "effective_retention": "30d",                           
        "retention_determined_by": "data_stream_configuration"  
      }
    }
  ]
}

Retention is applied to the remaining backing indices of a data stream as the last step of a data stream lifecycle run. Data stream lifecycle will retrieve the backing indices whose generation_time is longer than the effective retention period and delete them. The generation_time is only applicable to rolled over backing indices and it is either the time since the backing index got rolled over, or the time optionally configured in the index.lifecycle.origination_date setting.