Reduce storage

The amount of storage for APM data depends on several factors: how many services you are instrumenting, how much traffic the services see, agent and server settings, and for how long you keep monitoring data.

The transaction sample rate directly influences the number of documents (more precisely, spans) to be indexed and therefore is the most obvious way to reduce storage.

The transaction sample rate is a configuration setting of the agents.

Reducing it does not affect the collection of metrics such as Transactions Per Second.

Elastic APM agents collect stacktrace information under certain circumstances. This can be very helpful by identifying issues in your code, but it also comes with an overhead at collection time and increases the storage usage. Stacktrace collection settings are managed in the agents.

You might want to keep data only for a defined time period and delete old documents periodically, or you might also want to delete data collected for specific services or customers, or delete specific indices. Depending on your use case, you can either delete data periodically with Curator, by using the Delete By Query API, or by using the Kibana Index Management UI.

To delete data periodically you can use Curator and set up a cron job to run it.

By default APM indices have the pattern apm-%{[beat.version]}-{type}-%{+yyyy.MM.dd}. With the curator command line interface you can, for instance, see all your existing indices:

curator_cli --host localhost show_indices --filter_list '[\{"filtertype":"pattern","kind":"prefix","value":"apm-"\}]'

apm-6.8.11-error-2018.05.10
apm-6.8.11-error-2018.05.11
apm-6.8.11-error-2018.05.12
apm-6.8.11-sourcemap
apm-6.8.11-span-2018.05.10
apm-6.8.11-span-2018.05.11
apm-6.8.11-span-2018.05.12
apm-6.8.11-transaction-2018.05.10
apm-6.8.11-transaction-2018.05.11
apm-6.8.11-transaction-2018.05.12

And then delete any span indices older than 1 day:

curator_cli --host localhost delete_indices --filter_list '[\{"filtertype":"pattern","kind":"prefix","value":"apm-6.8.11-span-"\}, \{"filtertype":"age","source":"name","timestring":"%Y.%m.%d","unit":"days","unit_count":1,"direction":"older"\}]'

INFO      Deleting selected indices: [apm-6.8.11-span-2018.05.10, apm-6.8.11-span-2018.05.11]
INFO      ---deleting index apm-6.8.11-span-2018.05.10
INFO      ---deleting index apm-6.8.11-span-2018.05.11
INFO      "delete_indices" action completed.

To delete documents matching a specific query, for example, all documents with a given context.service.name, use the following request:

POST /apm-*/_delete_by_query
{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "context.service.name": {
              "value": "old-service-name"
            }
          }
        }
      ]
    }
  }
}

See delete by query for further information on this topic.

Select the indices you want to delete, and click the Manage indices button to see the available actions. Then click delete indices. Follow the Kibana Index Management documentation to get started with the index management UI.