Reduce storage

The amount of storage for APM data depends on several factors. Depending on how many services you are instrumenting and how much traffic the services see, the number of recorded transactions varies. You can influence the detail level of the information collected by reducing the sample rate or reducing collected stacktrace information. Another factor is for how long you want to keep monitoring data around.

In case you are monitoring high traffic services you might want to decrease the number of transactions that are sampled. The transaction sample rate directly influences the number of documents to be indexed and therefore is the most obvious way to reduce storage.

The transaction sample rate is controlled in the configuration of agents (for example for Python and Node.js).

Reducing the transaction sample rate does not affect the collection of metrics such as Transactions Per Second.

Elastic APM agents automatically collect information regarding stacktraces under certain circumstances. This can be very helpful by identifying issues in your code, but it also comes with an overhead at collection time and increaeses the storage usage. If you want to decrease collected stacktrace information you have to configure this directly in the agents. Please refer to our agent documentation to read more about the options in the single agents.

You might want to delete data for several reasons. A common use case is to keep data only for a defined time period and delete older documents. You might also want to delete data collected for specific services or customers, or delete specific indices. Depending on your use case, you can either delete data periodically with a tool like Curator or by using the Delete By Query API or by using the Kibana Index Management UI.

It might make sense to delete old APM indices on a periodic basis to make room for new data. To do this you can use a tool like Curator and set up a cron job to run it periodically.

By default APM indices have the pattern apm-%{[beat.version]}-{type}-%{+yyyy.MM.dd}. With the curator command line interface you can, for instance, see all your existing indices:

curator_cli --host localhost show_indices --filter_list '[\{"filtertype":"pattern","kind":"prefix","value":"apm-"\}]'

apm-6.3.2-error-2018.05.10
apm-6.3.2-error-2018.05.11
apm-6.3.2-error-2018.05.12
apm-6.3.2-sourcemap
apm-6.3.2-span-2018.05.10
apm-6.3.2-span-2018.05.11
apm-6.3.2-span-2018.05.12
apm-6.3.2-transaction-2018.05.10
apm-6.3.2-transaction-2018.05.11
apm-6.3.2-transaction-2018.05.12

And then delete any span indices older than 1 day:

curator_cli --host localhost delete_indices --filter_list '[\{"filtertype":"pattern","kind":"prefix","value":"apm-6.3.2-span-"\}, \{"filtertype":"age","source":"name","timestring":"%Y.%m.%d","unit":"days","unit_count":1,"direction":"older"\}]'

INFO      Deleting selected indices: [apm-6.3.2-span-2018.05.10, apm-6.3.2-span-2018.05.11]
INFO      ---deleting index apm-6.3.2-span-2018.05.10
INFO      ---deleting index apm-6.3.2-span-2018.05.11
INFO      "delete_indices" action completed.

In case you want to delete documents matching a specific query, e.g. all documents with a given context.service.name, you can do this by sending the following request:

POST /apm-*/_delete_by_query
{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "context.service.name": {
              "value": "old-service-name"
            }
          }
        }
      ]
    }
  }
}

See delete by query for further information on this topic.

Follow the Kibana Index Management docs for how to get started with the index management UI. Select the indices you want to delete, then click the Manage indices button to see supported options. Choose delete indices and your indices will be deleted.