Index lifecycle management (ILM)

The index lifecycle management (ILM) feature in Elasticsearch allows you to automate the lifecycle of your APM Server indices as they grow and age.

ILM is enabled by default in v7.3+, and the default ILM policy will be applied to all APM indices as long as all of the following conditions are true:

If you wish to define your own index lifecycle policies, see Custom configuration.

ILM manages an index based on its defined policy. The default ILM configuration applies hot and warm phases to all APM events: span, transaction, error, and metric. Cold and delete phases are not defined.

If you’d like APM Server to assist you in creating a custom index lifecycle policy, you must first ensure that ILM and ILM setup are enabled. You can then create a custom policy and map it to the correct APM event types. All of this can be done directly from the apm-server.yml file.

The following table can help you determine which options to configure when enabling, setting up, or disabling ILM:

ILM is enabled/disabled with apm-server.ilm.enabled:

apm-server:
  ilm:
    enabled: "auto"

Can be "auto", "true", or "false". Defaults to "auto" in v7.3+.

ILM setup can be customized with the apm-server.ilm.setup configuration options. The default configuration enables APM Server to handle all of the setup required for ILM:

apm-server:
  ilm:
    setup:
      enabled: true
      overwrite: true
      require_policy: true
      mapping:
        - event_type: "error"
          policy_name: "apm-rollover-30-days"
        - event_type: "span"
          policy_name: "apm-rollover-30-days"
        - event_type: "transaction"
          policy_name: "apm-rollover-30-days"
        - event_type: "metric"
          policy_name: "apm-rollover-30-days"

Defaults to true.

When true, APM Server will create an ILM specific index template for each APM event type. This is required to map ILM aliases and policies to indices.

When false, ILM setup is disabled. No policies, templates, or aliases will be created by APM Server. Only disable ilm.setup if you want to set up index management on your own. If you simply want to disable ILM, use apm-server.ilm.enabled: false instead.

Defaults to false. When false, APM Server will not overwrite any existing policies or ILM related templates. When first setting up ILM, your initial template and policy will be applied. You must set this to true when customizing your policies and template for them to be applied, or if you want to switch between managed and unmanaged indices.

Defaults to true, which means that an ILM policy must be defined in apm-server.yml. Changing this to false allows you to manually set up ILM policies and templates outside of APM Server, e.g., with Elasticsearch APIs. APM Server will still make use of ILM and connect your template with the defined mapping.

Maps each event type to the named policy. APM event types can only be error, span, transaction, and metric. Policies defined must be mapped to an event type. If they are not, they will not be sent to Elasticsearch. If you attempt to map an index lifecycle policy to a different event type, APM Server will not start. If you only map a subset of APM event types, the default values will be used for omitted event types.

Policies only need to be created once and will persist through version upgrades. Any change in existing ILM policies will only take place once the next phase is entered. You can define as many policies as you’d like. Just make sure to include the policy name in the ilm.setup.mapping. If your policy isn’t mapped to an event type, it will not be sent to Elasticsearch.

APM Server doesn’t do any validation on policies. Instead, if something is incorrectly defined, Elasticsearch will respond with 400 and APM Server won’t connect.

The default ILM policy can be viewed and edited in two places:

Head on over to the Elasticsearch documentation to learn more about all available policy phases and actions.

After starting up APM Server, you can confirm the policy was created by using the GET lifecycle policy API:

GET _ilm/policy

Here’s what a custom ILM configuration might look like. The example below creates two different policies, one for errors and spans, and another for transactions and metrics.

The apm-error-span-policy applies all four phases to its index lifecycle, including a cold phase with frozen indices, and a delete phase after 30 days. The apm-transaction-metric-policy keeps data in the hot, warm, and cold phases for a longer period of time, and does not delete any data.

  ilm:
    enabled: "auto"
    setup:
      mapping:
        - event_type: "error"
          policy_name: "apm-error-span-policy"
        - event_type: "span"
          policy_name: "apm-error-span-policy"
        - event_type: "transaction"
          policy_name: "apm-transaction-metric-policy"
        - event_type: "metric"
          policy_name: "apm-transaction-metric-policy"
      enabled: true
      policies:
        - name: "apm-error-span-policy"
          policy:
            phases:
              hot:
                actions:
                  rollover:
                    max_size: "50gb"
                    max_age: "1d"
                  set_priority:
                    priority: 100
              warm:
                min_age: "7d"
                actions:
                  set_priority:
                    priority: 50
                  readonly: {}
              cold:
                min_age: "30d"
                actions:
                  set_priority:
                    priority: 0
                  freeze: {}
              delete:
                min_age: "60d"
                actions:
                  delete: {}
        - name: "apm-transaction-metric-policy"
          policy:
            phases:
              hot:
                actions:
                  rollover:
                    max_size: "50gb"
                    max_age: "30d"
                  set_priority:
                    priority: 100
              warm:
                min_age: "60d"
                actions:
                  set_priority:
                    priority: 50
                  readonly: {}
              cold:
                min_age: "90d"
                actions:
                  set_priority:
                    priority: 0
                  freeze: {}