This documentation contains work-in-progress information for future Elastic Stack and Cloud releases. Use the version selector to view supported release docs. It also contains some Elastic Cloud serverless information. Check out our serverless docs for more details.
« Detections requirements About detection rules »
Elastic Docs ›Elastic Security Solution [master] ›Detections and alerts

Using logsdb index mode with Elastic Security

edit

Using logsdb index mode with Elastic Security

edit

To use the synthetic _source feature, you must have the appropriate subscription. Refer to the subscription page for Elastic Cloud and Elastic Stack/self-managed for the breakdown of available features and their associated subscription tiers.

This topic explains the impact of using logsdb index mode with Elastic Security.

With logsdb index mode, the original _source field is not stored in the index but can be reconstructed using synthetic _source.

When the _source is reconstructed, modifications are possible. Therefore, there could be a mismatch between users' expectations and how fields are formatted.

Continue reading to find out how this affects specific Elastic Security components.

Logsdb is not recommended for Elastic Security at this time. Users must fully understand and accept the documented changes to detection alert documents (see below), and ensure their deployment has excess hot data tier CPU resource capacity before enabling logsdb mode, as logsdb mode requires additional CPU resources during the ingest/indexing process. Enabling logsdb without sufficient hot data tier CPU may result in data ingestion backups and/or security detection rule timeouts and errors.

Alerts

edit

When alerts are generated, the _source event is copied into the alert to retain the original data. When the logsdb index mode is applied, the _source event stored in the alert is reconstructed using synthetic _source.

If you’re switching to use logsdb index mode, the _source field stored in the alert might look different in certain situations:

Alerts generated by the following rule types could be affected:

  • Custom query
  • Event correlation (non-sequence only)
  • Non-aggregate rule types (for example, ES|QL rules that use non-aggregating queries)

Alerts that are generated by threshold, machine learning, and event correlation sequence rules are not affected since they do not contain copies of the original source.

Rule actions

edit

While we do not recommend using _source for actions, in cases where the action relies on the _source, the same limitations and changes apply.

If you send alert notifications by enabling actions to the external systems that have workflows or automations based on fields formatted from the original source, they may be affected. In particular, this can happen when the fields used are arrays of objects.

We recommend checking and adjusting the rule actions using _source before switching to logsdb index mode.

Runtime fields

edit

Runtime fields that reference _source may be affected. Some runtime fields might not work and need to be adjusted. For example, if an event was indexed with the value of agent.name in the dot-notation form, it will be returned in the nested form and might not work.

The following is an example of accessing _source that works with the logsdb index mode enabled:

"source": """  emit(params._source.agent.name + "_____" + doc['agent.name'].value ); """
"source": """  emit(params._source['agent']['name'] + "_____" + doc['agent.name'].value );  """
"source": """  emit(field('agent.name').get(null) + "_____" + doc['agent.name'].value ); """
"source": """  emit($('agent.name', null) + "_____" + doc['agent.name'].value ); """

The following will not work with synthetic source (logsdb index mode enabled):

"source": """  emit(params._source['agent.name'] + "_____" + doc['agent.name'].value );  """
« Detections requirements About detection rules »