This documentation contains work-in-progress information for future Elastic Stack and Cloud releases. Use the version selector to view supported release docs. It also contains some Elastic Cloud serverless information. Check out our serverless docs for more details.

« Time series data stream (TSDS) Time series index settings »

› › ›

Set up a time series data stream (TSDS)

edit

Set up a time series data stream (TSDS)

edit

To set up a time series data stream (TSDS), follow these steps:

Prerequisites

edit

Before you create a TSDS, you should be familiar with data streams and TSDS concepts.
To follow this tutorial, you must have the following permissions:
- Cluster privileges: manage_ilm and manage_index_templates.
- Index privileges: create_doc and create_index for any TSDS you create or convert. To roll over a TSDS, you must have the manage privilege.

Create an index lifecycle policy

edit

While optional, we recommend using ILM to automate the management of your TSDS’s backing indices. ILM requires an index lifecycle policy.

We recommend you specify a max_age criteria for the rollover action in the policy. This ensures the @timestamp ranges for the TSDS’s backing indices are consistent. For example, setting a max_age of 1d for the rollover action ensures your backing indices consistently contain one day’s worth of data.

resp = client.ilm.put_lifecycle(
    name="my-weather-sensor-lifecycle-policy",
    policy={
        "phases": {
            "hot": {
                "actions": {
                    "rollover": {
                        "max_age": "1d",
                        "max_primary_shard_size": "50gb"
                    }
                }
            },
            "warm": {
                "min_age": "30d",
                "actions": {
                    "shrink": {
                        "number_of_shards": 1
                    },
                    "forcemerge": {
                        "max_num_segments": 1
                    }
                }
            },
            "cold": {
                "min_age": "60d",
                "actions": {
                    "searchable_snapshot": {
                        "snapshot_repository": "found-snapshots"
                    }
                }
            },
            "frozen": {
                "min_age": "90d",
                "actions": {
                    "searchable_snapshot": {
                        "snapshot_repository": "found-snapshots"
                    }
                }
            },
            "delete": {
                "min_age": "735d",
                "actions": {
                    "delete": {}
                }
            }
        }
    },
)
print(resp)

const response = await client.ilm.putLifecycle({
  name: "my-weather-sensor-lifecycle-policy",
  policy: {
    phases: {
      hot: {
        actions: {
          rollover: {
            max_age: "1d",
            max_primary_shard_size: "50gb",
          },
        },
      },
      warm: {
        min_age: "30d",
        actions: {
          shrink: {
            number_of_shards: 1,
          },
          forcemerge: {
            max_num_segments: 1,
          },
        },
      },
      cold: {
        min_age: "60d",
        actions: {
          searchable_snapshot: {
            snapshot_repository: "found-snapshots",
          },
        },
      },
      frozen: {
        min_age: "90d",
        actions: {
          searchable_snapshot: {
            snapshot_repository: "found-snapshots",
          },
        },
      },
      delete: {
        min_age: "735d",
        actions: {
          delete: {},
        },
      },
    },
  },
});
console.log(response);

PUT _ilm/policy/my-weather-sensor-lifecycle-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_primary_shard_size": "50gb"
          }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "60d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "found-snapshots"
          }
        }
      },
      "frozen": {
        "min_age": "90d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "found-snapshots"
          }
        }
      },
      "delete": {
        "min_age": "735d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Create an index template

edit

To setup a TSDS create an index template with the following details:

One or more index patterns that match the TSDS’s name. We recommend using our data stream naming scheme.
Enable data streams.
Specify a mapping that defines your dimensions and metrics:
- One or more dimension fields with a time_series_dimension value of true. Alternatively, one or more pass-through fields configured as dimension containers, provided that they will contain at least one sub-field (mapped statically or dynamically).
- One or more metric fields, marked using the time_series_metric mapping parameter.
- Optional: A date or date_nanos mapping for the @timestamp field. If you don’t specify a mapping, Elasticsearch maps @timestamp as a date field with default options.
Define index settings:
- Set index.mode setting to time_series.
- Your lifecycle policy in the index.lifecycle.name index setting.
- Optional: Other index settings, such as index.number_of_replicas, for your TSDS’s backing indices.
A priority higher than 200 to avoid collisions with built-in templates. See Avoid index pattern collisions.
Optional: Component templates containing your mappings and other index settings.

resp = client.indices.put_index_template(
    name="my-weather-sensor-index-template",
    index_patterns=[
        "metrics-weather_sensors-*"
    ],
    data_stream={},
    template={
        "settings": {
            "index.mode": "time_series",
            "index.lifecycle.name": "my-lifecycle-policy"
        },
        "mappings": {
            "properties": {
                "sensor_id": {
                    "type": "keyword",
                    "time_series_dimension": True
                },
                "location": {
                    "type": "keyword",
                    "time_series_dimension": True
                },
                "temperature": {
                    "type": "half_float",
                    "time_series_metric": "gauge"
                },
                "humidity": {
                    "type": "half_float",
                    "time_series_metric": "gauge"
                },
                "@timestamp": {
                    "type": "date"
                }
            }
        }
    },
    priority=500,
    meta={
        "description": "Template for my weather sensor data"
    },
)
print(resp)

response = client.indices.put_index_template(
  name: 'my-weather-sensor-index-template',
  body: {
    index_patterns: [
      'metrics-weather_sensors-*'
    ],
    data_stream: {},
    template: {
      settings: {
        'index.mode' => 'time_series',
        'index.lifecycle.name' => 'my-lifecycle-policy'
      },
      mappings: {
        properties: {
          sensor_id: {
            type: 'keyword',
            time_series_dimension: true
          },
          location: {
            type: 'keyword',
            time_series_dimension: true
          },
          temperature: {
            type: 'half_float',
            time_series_metric: 'gauge'
          },
          humidity: {
            type: 'half_float',
            time_series_metric: 'gauge'
          },
          "@timestamp": {
            type: 'date'
          }
        }
      }
    },
    priority: 500,
    _meta: {
      description: 'Template for my weather sensor data'
    }
  }
)
puts response

const response = await client.indices.putIndexTemplate({
  name: "my-weather-sensor-index-template",
  index_patterns: ["metrics-weather_sensors-*"],
  data_stream: {},
  template: {
    settings: {
      "index.mode": "time_series",
      "index.lifecycle.name": "my-lifecycle-policy",
    },
    mappings: {
      properties: {
        sensor_id: {
          type: "keyword",
          time_series_dimension: true,
        },
        location: {
          type: "keyword",
          time_series_dimension: true,
        },
        temperature: {
          type: "half_float",
          time_series_metric: "gauge",
        },
        humidity: {
          type: "half_float",
          time_series_metric: "gauge",
        },
        "@timestamp": {
          type: "date",
        },
      },
    },
  },
  priority: 500,
  _meta: {
    description: "Template for my weather sensor data",
  },
});
console.log(response);

PUT _index_template/my-weather-sensor-index-template
{
  "index_patterns": ["metrics-weather_sensors-*"],
  "data_stream": { },
  "template": {
    "settings": {
      "index.mode": "time_series",
      "index.lifecycle.name": "my-lifecycle-policy"
    },
    "mappings": {
      "properties": {
        "sensor_id": {
          "type": "keyword",
          "time_series_dimension": true
        },
        "location": {
          "type": "keyword",
          "time_series_dimension": true
        },
        "temperature": {
          "type": "half_float",
          "time_series_metric": "gauge"
        },
        "humidity": {
          "type": "half_float",
          "time_series_metric": "gauge"
        },
        "@timestamp": {
          "type": "date"
        }
      }
    }
  },
  "priority": 500,
  "_meta": {
    "description": "Template for my weather sensor data"
  }
}

Create the TSDS

edit

Indexing requests add documents to a TSDS. Documents in a TSDS must include:

A @timestamp field
One or more dimension fields. At least one dimension must match the index.routing_path index setting, if specified. If not specified explicitly, index.routing_path is set automatically to whichever mappings have time_series_dimension set to true.

To automatically create your TSDS, submit an indexing request that targets the TSDS’s name. This name must match one of your index template’s index patterns.

To test the following example, update the timestamps to within three hours of your current time. Data added to a TSDS must always fall within an accepted time range.

resp = client.bulk(
    index="metrics-weather_sensors-dev",
    operations=[
        {
            "create": {}
        },
        {
            "@timestamp": "2099-05-06T16:21:15.000Z",
            "sensor_id": "HAL-000001",
            "location": "plains",
            "temperature": 26.7,
            "humidity": 49.9
        },
        {
            "create": {}
        },
        {
            "@timestamp": "2099-05-06T16:25:42.000Z",
            "sensor_id": "SYKENET-000001",
            "location": "swamp",
            "temperature": 32.4,
            "humidity": 88.9
        }
    ],
)
print(resp)

resp1 = client.index(
    index="metrics-weather_sensors-dev",
    document={
        "@timestamp": "2099-05-06T16:21:15.000Z",
        "sensor_id": "SYKENET-000001",
        "location": "swamp",
        "temperature": 32.4,
        "humidity": 88.9
    },
)
print(resp1)

const response = await client.bulk({
  index: "metrics-weather_sensors-dev",
  operations: [
    {
      create: {},
    },
    {
      "@timestamp": "2099-05-06T16:21:15.000Z",
      sensor_id: "HAL-000001",
      location: "plains",
      temperature: 26.7,
      humidity: 49.9,
    },
    {
      create: {},
    },
    {
      "@timestamp": "2099-05-06T16:25:42.000Z",
      sensor_id: "SYKENET-000001",
      location: "swamp",
      temperature: 32.4,
      humidity: 88.9,
    },
  ],
});
console.log(response);

const response1 = await client.index({
  index: "metrics-weather_sensors-dev",
  document: {
    "@timestamp": "2099-05-06T16:21:15.000Z",
    sensor_id: "SYKENET-000001",
    location: "swamp",
    temperature: 32.4,
    humidity: 88.9,
  },
});
console.log(response1);

PUT metrics-weather_sensors-dev/_bulk
{ "create":{ } }
{ "@timestamp": "2099-05-06T16:21:15.000Z", "sensor_id": "HAL-000001", "location": "plains", "temperature": 26.7,"humidity": 49.9 }
{ "create":{ } }
{ "@timestamp": "2099-05-06T16:25:42.000Z", "sensor_id": "SYKENET-000001", "location": "swamp", "temperature": 32.4, "humidity": 88.9 }

POST metrics-weather_sensors-dev/_doc
{
  "@timestamp": "2099-05-06T16:21:15.000Z",
  "sensor_id": "SYKENET-000001",
  "location": "swamp",
  "temperature": 32.4,
  "humidity": 88.9
}

You can also manually create the TSDS using the create data stream API. The TSDS’s name must still match one of your template’s index patterns.

resp = client.indices.create_data_stream(
    name="metrics-weather_sensors-dev",
)
print(resp)

response = client.indices.create_data_stream(
  name: 'metrics-weather_sensors-dev'
)
puts response

const response = await client.indices.createDataStream({
  name: "metrics-weather_sensors-dev",
});
console.log(response);

PUT _data_stream/metrics-weather_sensors-dev

Secure the TSDS

edit

Use index privileges to control access to a TSDS. Granting privileges on a TSDS grants the same privileges on its backing indices.

For an example, refer to Data stream privileges.

Convert an existing data stream to a TSDS

edit

You can also use the above steps to convert an existing regular data stream to a TSDS. In this case, you’ll want to:

Edit your existing index lifecycle policy, component templates, and index templates instead of creating new ones.

Instead of creating the TSDS, manually roll over its write index. This ensures the current write index and any new backing indices have an index.mode of time_series.

You can manually roll over the write index using the rollover API.

resp = client.indices.rollover(
    alias="metrics-weather_sensors-dev",
)
print(resp)

response = client.indices.rollover(
  alias: 'metrics-weather_sensors-dev'
)
puts response

const response = await client.indices.rollover({
  alias: "metrics-weather_sensors-dev",
});
console.log(response);

POST metrics-weather_sensors-dev/_rollover

A note about component templates and index.mode setting

edit

Configuring a TSDS via an index template that uses component templates is a bit more complicated. Typically with component templates mappings and settings get scattered across multiple component templates. If the index.routing_path is defined, the fields it references need to be defined in the same component template with the time_series_dimension attribute enabled.

The reasons for this is that each component template needs to be valid on its own. When configuring the index.mode setting in an index template, the index.routing_path setting is configured automatically. It is derived from the field mappings with time_series_dimension attribute enabled.

What’s next?

edit

Now that you’ve set up your TSDS, you can manage and use it like a regular data stream. For more information, refer to:

« Time series data stream (TSDS) Time series index settings »