Set up a time series data stream (TSDS)
editSet up a time series data stream (TSDS)
editTo set up a time series data stream (TSDS), follow these steps:
Prerequisites
edit- Before you create a TSDS, you should be familiar with data streams and TSDS concepts.
-
To follow this tutorial, you must have the following permissions:
-
Cluster privileges:
manage_ilm
andmanage_index_templates
. -
Index privileges:
create_doc
andcreate_index
for any TSDS you create or convert. To roll over a TSDS, you must have themanage
privilege.
-
Cluster privileges:
Create an index lifecycle policy
editWhile optional, we recommend using ILM to automate the management of your TSDS’s backing indices. ILM requires an index lifecycle policy.
We recommend you specify a max_age
criteria for the rollover
action in the
policy. This ensures the @timestamp
ranges for the
TSDS’s backing indices are consistent. For example, setting a max_age
of 1d
for the rollover
action ensures your backing indices consistently contain one
day’s worth of data.
PUT _ilm/policy/my-weather-sensor-lifecycle-policy { "policy": { "phases": { "hot": { "actions": { "rollover": { "max_age": "1d", "max_primary_shard_size": "50gb" } } }, "warm": { "min_age": "30d", "actions": { "shrink": { "number_of_shards": 1 }, "forcemerge": { "max_num_segments": 1 } } }, "cold": { "min_age": "60d", "actions": { "searchable_snapshot": { "snapshot_repository": "found-snapshots" } } }, "frozen": { "min_age": "90d", "actions": { "searchable_snapshot": { "snapshot_repository": "found-snapshots" } } }, "delete": { "min_age": "735d", "actions": { "delete": {} } } } } }
Create a mappings component template
editA TSDS requires a matching index template. In most cases, you compose this index template using one or more component templates. You typically use separate component templates for mappings and index settings. This lets you reuse the component templates in multiple index templates.
For a TSDS, the mappings component template must include mappings for:
-
One or more dimension fields with a
time_series_dimension
value oftrue
. At least one of these dimensions must be a plainkeyword
field.
Optionally, the template can also include mappings for:
-
One or more metric fields, marked using the
time_series_metric
mapping parameter. -
A
date
ordate_nanos
mapping for the@timestamp
field. If you don’t specify a mapping, Elasticsearch maps@timestamp
as adate
field with default options.
response = client.cluster.put_component_template( name: 'my-weather-sensor-mappings', body: { template: { mappings: { properties: { sensor_id: { type: 'keyword', time_series_dimension: true }, location: { type: 'keyword', time_series_dimension: true }, temperature: { type: 'half_float', time_series_metric: 'gauge' }, humidity: { type: 'half_float', time_series_metric: 'gauge' }, "@timestamp": { type: 'date', format: 'strict_date_optional_time' } } } }, _meta: { description: 'Mappings for weather sensor data' } } ) puts response
PUT _component_template/my-weather-sensor-mappings { "template": { "mappings": { "properties": { "sensor_id": { "type": "keyword", "time_series_dimension": true }, "location": { "type": "keyword", "time_series_dimension": true }, "temperature": { "type": "half_float", "time_series_metric": "gauge" }, "humidity": { "type": "half_float", "time_series_metric": "gauge" }, "@timestamp": { "type": "date", "format": "strict_date_optional_time" } } } }, "_meta": { "description": "Mappings for weather sensor data" } }
Create an index settings component template
editOptionally, the index settings component template for a TSDS can include:
-
Your lifecycle policy in the
index.lifecycle.name
index setting. -
The
index.look_ahead_time
index setting. -
Other index settings, such as
index.codec
, for your TSDS’s backing indices.
Don’t specify the index.routing_path
index setting in a component
template. If you choose, you can configure index.routing_path
directly in the
index template, as shown in the following step.
response = client.cluster.put_component_template( name: 'my-weather-sensor-settings', body: { template: { settings: { "index.lifecycle.name": 'my-lifecycle-policy', "index.look_ahead_time": '3h', "index.codec": 'best_compression' } }, _meta: { description: 'Index settings for weather sensor data' } } ) puts response
PUT _component_template/my-weather-sensor-settings { "template": { "settings": { "index.lifecycle.name": "my-lifecycle-policy", "index.look_ahead_time": "3h", "index.codec": "best_compression" } }, "_meta": { "description": "Index settings for weather sensor data" } }
Create an index template
editUse your component templates to create an index template. In the index template, specify:
- One or more index patterns that match the TSDS’s name. We recommend using our data stream naming scheme.
- That the template is data stream enabled.
-
An
index.mode
object set totime_series
. -
Optional: The
index.routing_path
index setting. The setting value should only match plainkeyword
dimension fields and should be set directly in the index template. When not defined explicitly, theindex.routing_path
setting is generated from the mapping using all mappings that are set withtime_series_dimension
set totrue
. - The component templates containing your mappings and other index settings.
-
A priority higher than
200
to avoid collisions with built-in templates. See Avoid index pattern collisions.
response = client.indices.put_index_template( name: 'my-weather-sensor-index-template', body: { index_patterns: [ 'metrics-weather_sensors-*' ], data_stream: {}, template: { settings: { "index.mode": 'time_series', "index.routing_path": [ 'sensor_id', 'location' ] } }, composed_of: [ 'my-weather-sensor-mappings', 'my-weather-sensor-settings' ], priority: 500, _meta: { description: 'Template for my weather sensor data' } } ) puts response
PUT _index_template/my-weather-sensor-index-template { "index_patterns": ["metrics-weather_sensors-*"], "data_stream": { }, "template": { "settings": { "index.mode": "time_series", "index.routing_path": [ "sensor_id", "location" ] } }, "composed_of": [ "my-weather-sensor-mappings", "my-weather-sensor-settings" ], "priority": 500, "_meta": { "description": "Template for my weather sensor data" } }
Create the TSDS
editIndexing requests add documents to a TSDS. Documents in a TSDS must include:
-
A
@timestamp
field -
One or more dimension fields. At least one dimension must be a
keyword
field that matches theindex.routing_path
index setting, if specified. If not specified explicitly,index.routing_path
is set automatically to whichever mappings havetime_series_dimension
set totrue
.
To automatically create your TSDS, submit an indexing request that targets the TSDS’s name. This name must match one of your index template’s index patterns.
To test the following example, update the timestamps to within three hours of your current time. Data added to a TSDS must always fall within an accepted time range.
PUT metrics-weather_sensors-dev/_bulk { "create":{ } } { "@timestamp": "2099-05-06T16:21:15.000Z", "sensor_id": "HAL-000001", "location": "plains", "temperature": 26.7,"humidity": 49.9 } { "create":{ } } { "@timestamp": "2099-05-06T16:25:42.000Z", "sensor_id": "SYKENET-000001", "location": "swamp", "temperature": 32.4, "humidity": 88.9 } POST metrics-weather_sensors-dev/_doc { "@timestamp": "2099-05-06T16:21:15.000Z", "sensor_id": "SYKENET-000001", "location": "swamp", "temperature": 32.4, "humidity": 88.9 }
You can also manually create the TSDS using the create data stream API. The TSDS’s name must still match one of your template’s index patterns.
PUT _data_stream/metrics-weather_sensors-dev
Secure the TSDS
editUse index privileges to control access to a TSDS. Granting privileges on a TSDS grants the same privileges on its backing indices.
For an example, refer to Data stream privileges.
Convert an existing data stream to a TSDS
editYou can also use the above steps to convert an existing regular data stream to a TSDS. In this case, you’ll want to:
- Edit your existing index lifecycle policy, component templates, and index templates instead of creating new ones.
-
Instead of creating the TSDS, manually roll over its write index. This ensures the current write index and any new backing indices have an
index.mode
oftime_series
.You can manually roll over the write index using the rollover API.
response = client.indices.rollover( alias: 'metrics-weather_sensors-dev' ) puts response
POST metrics-weather_sensors-dev/_rollover
What’s next?
editNow that you’ve set up your TSDS, you can manage and use it like a regular data stream. For more information, refer to: