Logs data stream

A logs data stream is a data stream type that stores log data more efficiently.

In benchmarks, log data stored in a logs data stream used ~2.5 times less disk space than a regular data stream. The exact impact will vary depending on your data set.

To create a logs data stream, set your index template index.mode to logsdb:

resp = client.indices.put_index_template(
    name="my-index-template",
    index_patterns=[
        "logs-*"
    ],
    data_stream={},
    template={
        "settings": {
            "index.mode": "logsdb"
        }
    },
    priority=101,
)
print(resp)

const response = await client.indices.putIndexTemplate({
  name: "my-index-template",
  index_patterns: ["logs-*"],
  data_stream: {},
  template: {
    settings: {
      "index.mode": "logsdb",
    },
  },
  priority: 101,
});
console.log(response);

PUT _index_template/my-index-template
{
  "index_patterns": ["logs-*"],
  "data_stream": { },
  "template": {
     "settings": {
        "index.mode": "logsdb" 
     }
  },
  "priority": 101 
}

After the index template is created, new indices that use the template will be configured as a logs data stream. You can start indexing data and using the data stream.

By default, logsdb mode uses synthetic source, which omits storing the original _source field and synthesizes it from doc values or stored fields upon document retrieval. Synthetic source comes with a few restrictions which you can read more about in the documentation section dedicated to it.

For more details on this setting and ways to refine or bypass it, check out this section.

The following settings are applied by default when using the logsdb mode for index sorting:

logsdb index mode allows users to override the default sort settings. For instance, users can specify their own fields and order for sorting by modifying the index.sort.field and index.sort.order.

When using default sort settings, the host.name field is automatically injected into the mappings of the index as a keyword field to ensure that sorting can be applied. This guarantees that logs are efficiently sorted and retrieved based on the host.name and @timestamp fields.

Once an index is created, the sort settings are immutable and cannot be modified. To apply different sort settings, a new index must be created with the desired configuration. For data streams, this can be achieved by means of an index rollover after updating relevant (component) templates.

If the default sort settings are not suitable for your use case, consider modifying them. Keep in mind that sort settings can influence indexing throughput, query latency, and may affect compression efficiency due to the way data is organized after sorting. For more details, refer to our documentation on index sorting.

logsdb index mode uses the best_compression codec by default, which applies ZSTD compression to stored fields. Users are allowed to override it and switch to the default codec for faster compression at the expense of slightly larger storage footprint.

logsdb index mode also adopts specialized codecs for numeric doc values that are crafted to optimize storage usage. Users can rely on these specialized codecs being applied by default when using logsdb index mode.

Doc values encoding for numeric fields in logsdb follows a static sequence of codecs, applying each one in the following order: delta encoding, offset encoding, Greatest Common Divisor GCD encoding, and finally Frame Of Reference (FOR) encoding. The decision to apply each encoding is based on heuristics determined by the data distribution. For example, before applying delta encoding, the algorithm checks if the data is monotonically non-decreasing or non-increasing. If the data fits this pattern, delta encoding is applied; otherwise, the next encoding is considered.

The encoding is specific to each Lucene segment and is also re-applied at segment merging time. The merged Lucene segment may use a different encoding compared to the original Lucene segments, based on the characteristics of the merged data.

The following methods are applied sequentially:

For keyword fields, Run Length Encoding (RLE) is applied to the ordinals, which represent positions in the Lucene segment-level keyword dictionary. This compression is used when multiple consecutive documents share the same keyword.

By default, logsdb index mode sets ignore_malformed to true. This setting allows documents with malformed fields to be indexed without causing indexing failures, ensuring that log data ingestion continues smoothly even when some fields contain invalid or improperly formatted data.

Users can override this setting by setting index.mapping.ignore_malformed to false. However, this is not recommended as it might result in documents with malformed fields being rejected and not indexed at all.

In logsdb index mode, the index.mapping.ignore_above setting is applied by default at the index level to ensure efficient storage and indexing of large keyword fields.The index-level default for ignore_above is set to 8191 characters. If using UTF-8 encoding, this results in a limit of 32764 bytes, depending on character encoding. The mapping-level ignore_above setting still takes precedence. If a specific field has an ignore_above value defined in its mapping, that value will override the index-level index.mapping.ignore_above value. This default behavior helps to optimize indexing performance by preventing excessively large string values from being indexed, while still allowing users to customize the limit, overriding it at the mapping level or changing the index level default setting.

In logsdb index mode, the setting index.mapping.total_fields.ignore_dynamic_beyond_limit is set to true by default. This allows dynamically mapped fields to be added on top of statically defined fields without causing document rejection, even after the total number of fields exceeds the limit defined by index.mapping.total_fields.limit. The index.mapping.total_fields.limit setting specifies the maximum number of fields an index can have (static, dynamic and runtime). When the limit is reached, new dynamically mapped fields will be ignored instead of failing the document indexing, ensuring continued log ingestion without errors.

When logsdb index mode uses synthetic _source, and doc_values are disabled for a field in the mapping, Elasticsearch may set the store setting to true for that field as a last resort option to ensure that the field’s data is still available for reconstructing the document’s source when retrieving it via synthetic source.

For example, this happens with text fields when store is false and there is no suitable multi-field available to reconstruct the original value in synthetic source.

This automatic adjustment allows synthetic source to work correctly, even when doc values are not enabled for certain fields.

The following is a summary of key settings that apply when using logsdb index mode in Elasticsearch: