Manual index lifecycle management

Set up the default index template.

If you haven’t already, you’ll need to set up the default index template. This is accomplished by running the setup --template command.

./apm-server setup --template

Create a policy for spans.

Index lifecycle management will manage an index based on its defined policy. Policies only need to be created once, and will persist through version upgrades. Let’s create a policy named apm_span_policy.

This policy defines two rollover criteria: "max_age": "1d", and "max_size": "50gb". When one or more of these criteria are met, the policy rolls data over into a new index. apm_span_policy also tells the old indexes to delete after 10 days. All of these values can be customized to your specific needs.

PUT _ilm/policy/apm_span_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "1d", 
            "max_size": "50gb" 
          }
        }
      },
      "delete": {
        "min_age": "10d", 
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Set up the ILM index template.

To use the index lifecycle management policy created in the previous step, you need to specify it in the index template used to create the indices. The following template associates apm_span_policy with indices created from the apm-7.3.2-span-ilm template.

PUT _template/apm-7.3.2-span-ilm
{
  "order": 2,
  "index_patterns": ["apm-7.3.2-span-*"], 
  "settings": {
    "index.lifecycle.name": "apm_span_policy", 
    "index.lifecycle.rollover_alias": "apm-7.3.2-span"
  }
}

Create the index and alias.

Now we can create the first index: apm-7.3.2-span-000001.

PUT apm-7.3.2-span-000001 
{
  "aliases": {
    "apm-7.3.2-span":{
      "is_write_index": true 
    }
  }
}

Verify the ILM index template was applied.

GET apm-*-span/_settings

The response should be similar to this:

{
  "apm-7.3.2-span-000001" : {
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "apm_span_policy",
          "rollover_alias" : "apm-7.3.2-span"
        },
        "number_of_shards" : "1",
        "provided_name" : "apm-7.3.2-span-000001",
        "creation_date" : "1553024227938",
        "number_of_replicas" : "1",
        "uuid" : "6b5l-H7QTRK95FAodAN-wg",
        "version" : {
          "created" : "7000099"
        }
      }
    }
  }
}

You can also verify and adjust your configuration via Kibana’s management.

Repeat for other indices.

Repeat the previous steps for each index that will be using ILM.

Modify APM Server’s configuration.

Finally, modify the default index configuration in apm-server.yml. Trim off the date template from each index you are setting up ILM for, so that APM Server is always writing events to the same place. The name of your modified index configuration must match the is_write_index alias created previously

It’s important to note that apm-server.yml overwrites defaults rather than being merged. This means you’ll need to configure all of your indices in the file, even if some are not using ILM.

output.elasticsearch:
  indices:
    - index: "apm-7.3.2-sourcemap"
      when.contains:
        processor.event: "sourcemap"

    - index: "apm-7.3.2-error"
      when.contains:
        processor.event: "error"

    - index: "apm-7.3.2-transaction"
      when.contains:
        processor.event: "transaction"

    - index: "apm-7.3.2-span"
      when.contains:
        processor.event: "span"

    - index: "apm-7.3.2-metric"
      when.contains:
        processor.event: "metric"

    - index: "apm-7.3.2-onboarding"
      when.contains:
        processor.event: "onboarding"

Start apm-server.

Your ILM configuration should now be up and running!