A newer version is available. For the latest information, see the current release documentation.
« Manage storage Index lifecycle management »
Elastic Docs › ›Manage storage

Data streams

edit

Data streams

edit

Elastic Agent uses data streams to store append-only time series data across multiple indices. Data streams are well-suited for logs, metrics, traces, and other continuously generated data, and offer a host of benefits over other indexing strategies:

  • Reduced number of fields per index
  • More granular data control
  • Flexible naming scheme
  • Fewer ingest permissions required

See the Fleet and Elastic Agent Guide to learn more.

Data stream naming scheme

edit

APM data follows the <type>-<dataset>-<namespace> naming scheme. The type and dataset are predefined by the APM integration, but the namespace is your opportunity to customize how different types of data are stored in Elasticsearch. There is no recommendation for what to use as your namespace—​it is intentionally flexible. For example, you might create namespaces for each of your environments, like dev, prod, production, etc. Or, you might create namespaces that correspond to strategic business units within your organization.

APM data streams

edit

By type, the APM data streams are:

Traces

Traces are comprised of spans and transactions. Traces are stored in the following data streams:

  • Application traces: traces-apm-<namespace>
Metrics

Metrics include application-based metrics and basic system metrics. Metrics are stored in the following data streams:

  • APM internal metrics: metrics-apm.internal-<namespace>
  • APM profiling metrics: metrics-apm.profiling-<namespace>

  • Application metrics: metrics-apm.app.<service.name>-<namespace>

    Application metrics include the instrumented service’s name—​defined in each APM agent’s configuration—​in the data stream name. Service names therefore must follow certain index naming rules.

    Service name rules
    • Service names are case-insensitive and must be unique. For example, you cannot have a service named Foo and another named foo.

    • Special characters will be removed from service names and replaced with underscores (_). Special characters include:

      '\\', '/', '*', '?', '"', '<', '>', '|', ' ', ',', '#', ':', '-'
Logs

Logs include application error events and application logs. Logs are stored in the following data streams:

  • APM error/exception logging: logs-apm.error-<namespace>

What’s next?

edit
  • Data streams define not only how data is stored in Elasticsearch, but also how data is retained over time. See Index lifecycle management to learn how to create your own data retention policies.
  • See Manage storage for information on APM storage and processing costs, processing and performance, and other index management features.
« Manage storage Index lifecycle management »