Logs data stream

Logs data streams and the logsdb index mode are in tech preview and may be changed or removed in the future. Don’t use logs data streams or logsdb index mode in production.

A logs data stream is a data stream type that stores log data more efficiently.

In benchmarks, log data stored in a logs data stream used ~2.5 times less disk space than a regular data stream. The exact impact will vary depending on your data set.

The following features are enabled in a logs data stream:

Synthetic source, which omits storing the _source field. When the document source is requested, it is synthesized from document fields upon retrieval.
Index sorting. This yields a lower storage footprint. By default indices are sorted by host.name and @timestamp fields at index time.
More space efficient compression for fields with doc_values enabled.

Create a logs data stream

edit

To create a logs data stream, set your index template index.mode to logsdb:

resp = client.indices.put_index_template(
    name="my-index-template",
    index_patterns=[
        "logs-*"
    ],
    data_stream={},
    template={
        "settings": {
            "index.mode": "logsdb"
        }
    },
    priority=101,
)
print(resp)

const response = await client.indices.putIndexTemplate({
  name: "my-index-template",
  index_patterns: ["logs-*"],
  data_stream: {},
  template: {
    settings: {
      "index.mode": "logsdb",
    },
  },
  priority: 101,
});
console.log(response);

PUT _index_template/my-index-template
{
  "index_patterns": ["logs-*"],
  "data_stream": { },
  "template": {
     "settings": {
        "index.mode": "logsdb" 
     }
  },
  "priority": 101 
}

	The index mode setting.
	The index template priority. By default, Elasticsearch ships with an index template with a `logs--` pattern with a priority of 100. You need to define a priority higher than 100 to ensure that this index template gets selected over the default index template for the `logs--` pattern. See the avoid index pattern collision section for more information.

After the index template is created, new indices that use the template will be configured as a logs data stream. You can start indexing data and using the data stream.

« Reindex a time series data stream (TSDS) Data stream lifecycle »