This documentation contains work-in-progress information for future Elastic Stack and Cloud releases. Use the version selector to view supported release docs. It also contains some Elastic Cloud serverless information. Check out our serverless docs for more details.
« Reindex a time series data stream (TSDS) Data stream lifecycle »
Elastic Docs ›Elasticsearch Guide [master] ›Data streams

Logs data stream

edit

Logs data stream

edit

Logs data streams and the logsdb index mode are in tech preview and may be changed or removed in the future. Don’t use logs data streams or logsdb index mode in production.

A logs data stream is a data stream type that stores log data more efficiently.

In benchmarks, log data stored in a logs data stream used ~2.5 times less disk space than a regular data stream. The exact impact will vary depending on your data set.

The following features are enabled in a logs data stream:

  • Synthetic source, which omits storing the _source field. When the document source is requested, it is synthesized from document fields upon retrieval.
  • Index sorting. This yields a lower storage footprint. By default indices are sorted by host.name and @timestamp fields at index time.
  • More space efficient compression for fields with doc_values enabled.

Create a logs data stream

edit

To create a logs data stream, set your index template index.mode to logsdb:

resp = client.indices.put_index_template(
    name="my-index-template",
    index_patterns=[
        "logs-*"
    ],
    data_stream={},
    template={
        "settings": {
            "index.mode": "logsdb"
        }
    },
    priority=101,
)
print(resp)
const response = await client.indices.putIndexTemplate({
  name: "my-index-template",
  index_patterns: ["logs-*"],
  data_stream: {},
  template: {
    settings: {
      "index.mode": "logsdb",
    },
  },
  priority: 101,
});
console.log(response);
PUT _index_template/my-index-template
{
  "index_patterns": ["logs-*"],
  "data_stream": { },
  "template": {
     "settings": {
        "index.mode": "logsdb" 
     }
  },
  "priority": 101 
}

The index mode setting.

The index template priority. By default, Elasticsearch ships with an index template with a logs-*-* pattern with a priority of 100. You need to define a priority higher than 100 to ensure that this index template gets selected over the default index template for the logs-*-* pattern. See the avoid index pattern collision section for more information.

After the index template is created, new indices that use the template will be configured as a logs data stream. You can start indexing data and using the data stream.

« Reindex a time series data stream (TSDS) Data stream lifecycle »