Tutorial: Update existing data stream

edit

Tutorial: Update existing data stream

edit

To update the lifecycle of an existing data stream you do the following actions:

Set a data stream’s lifecycle

edit

To add or to change the retention period of your data stream you can use the PUT lifecycle API.

You can set infinite retention period, meaning that your data should never be deleted. For example:

resp = client.indices.put_data_lifecycle(
    name="my-data-stream",
)
print(resp)

response = client.indices.put_data_lifecycle(
  name: 'my-data-stream',
  body: {}
)
puts response

const response = await client.indices.putDataLifecycle({
  name: "my-data-stream",
});
console.log(response);

PUT _data_stream/my-data-stream/_lifecycle
{ }

An empty payload means that your data stream is still managed but the data will never be deleted. Managing a time series data stream such as logs or metrics enables Elasticsearch to better store your data even if you do not use a retention period.

Or you can set the retention period of your choice. For example:

resp = client.indices.put_data_lifecycle(
    name="my-data-stream",
    data_retention="30d",
)
print(resp)

response = client.indices.put_data_lifecycle(
  name: 'my-data-stream',
  body: {
    data_retention: '30d'
  }
)
puts response

const response = await client.indices.putDataLifecycle({
  name: "my-data-stream",
  data_retention: "30d",
});
console.log(response);

PUT _data_stream/my-data-stream/_lifecycle
{
  "data_retention": "30d" 
}

The retention period of this data stream is set to 30 days. This means that Elasticsearch is allowed to delete data that is older than 30 days at its own discretion.

The changes in the lifecycle are applied on all backing indices of the data stream. You can see the effect of the change via the explain API:

resp = client.indices.explain_data_lifecycle(
    index=".ds-my-data-stream-*",
)
print(resp)

response = client.indices.explain_data_lifecycle(
  index: '.ds-my-data-stream-*'
)
puts response

const response = await client.indices.explainDataLifecycle({
  index: ".ds-my-data-stream-*",
});
console.log(response);

GET .ds-my-data-stream-*/_lifecycle/explain

The response will look like:

{
  "indices": {
    ".ds-my-data-stream-2023.04.19-000002": {
      "index": ".ds-my-data-stream-2023.04.19-000002",  
      "managed_by_lifecycle": true,                           
      "index_creation_date_millis": 1681919221417,
      "time_since_index_creation": "6.85s",             
      "lifecycle": {
        "enabled": true,
        "data_retention": "30d"                         
      }
    },
    ".ds-my-data-stream-2023.04.17-000001": {
      "index": ".ds-my-data-stream-2023.04.17-000001",  
      "managed_by_lifecycle": true,                           
      "index_creation_date_millis": 1681745209501,
      "time_since_index_creation": "48d",               
      "rollover_date_millis": 1681919221419,
      "time_since_rollover": "6.84s",                   
      "generation_time": "6.84s",                       
      "lifecycle": {
        "enabled": true,
        "data_retention": "30d"                         
      }
    }
  }
}

	The name of the backing index.
	This index is managed by a data stream lifecycle.
	The time that has passed since this index has been created.
	The data retention for this index is at least 30 days, as it was recently updated.
	The name of the backing index.
	This index is managed by the built-in data stream lifecycle.
	The time that has passed since this index has been created.
	The time that has passed since this index was rolled over.
	The time that will be used to determine when it’s safe to delete this index and all its data.
	The data retention for this index as well is at least 30 days, as it was recently updated.

Remove lifecycle for a data stream

edit

To remove the lifecycle of a data stream you can use the delete lifecycle API. As consequence, the maintenance operations that were applied by the lifecycle will no longer be applied to the data stream and all its backing indices. For example:

resp = client.indices.delete_data_lifecycle(
    name="my-data-stream",
)
print(resp)

response = client.indices.delete_data_lifecycle(
  name: 'my-data-stream'
)
puts response

const response = await client.indices.deleteDataLifecycle({
  name: "my-data-stream",
});
console.log(response);

DELETE _data_stream/my-data-stream/_lifecycle

You can then use the explain API again to see that the indices are no longer managed.

resp = client.indices.explain_data_lifecycle(
    index=".ds-my-data-stream-*",
)
print(resp)

response = client.indices.explain_data_lifecycle(
  index: '.ds-my-data-stream-*'
)
puts response

const response = await client.indices.explainDataLifecycle({
  index: ".ds-my-data-stream-*",
});
console.log(response);

GET .ds-my-data-stream-*/_lifecycle/explain

{
  "indices": {
    ".ds-my-data-stream-2023.04.19-000002": {
      "index": ".ds-my-data-stream-2023.04.19-000002",  
      "managed_by_lifecycle": false                           
    },
    ".ds-my-data-stream-2023.04.17-000001": {
      "index": ".ds-my-data-stream-2023.04.19-000001",  
      "managed_by_lifecycle": false                           
    }
  }
}

	The name of the backing index.
	Indication that the index is not managed by the data stream lifecycle.
	The name of another backing index.
	Indication that the index is not managed by the data stream lifecycle.

« Tutorial: Create a data stream with a lifecycle Tutorial: Migrate ILM managed data stream to data stream lifecycle »