NOTE: You are looking at documentation for an older release. For the latest information, see the current release documentation.

« Manually loading template configuration Configure logging »
Elastic Docs › ›Configuring APM Server

Manual index lifecycle management

edit

Manual index lifecycle management

edit

APM Server can take advantage of the index lifecycle management (ILM) capabilities of Elasticsearch. ILM enables you to automate how you want to manage your indices over time. You can base actions on factors such as shard size and performance requirements.

The guide below will help you set up a custom ILM policy for span indices. You can repeat the actions for any other indices.

If you’re migrating from an existing setup, any indices present before ILM was configured will need to be managed manually.

  1. Set up the default index template.

    If you haven’t already, you’ll need to set up the default index template. This is accomplished by running the setup --template command.

    ./apm-server setup --template

  2. Create a policy for spans.

    Index lifecycle management will manage an index based on its defined policy. Policies only need to be created once, and will persist through version upgrades. Let’s create a policy named apm_span_policy.

    This policy defines two rollover criteria: "max_age": "1d", and "max_size": "50gb". When one or more of these criteria are met, the policy rolls data over into a new index. apm_span_policy also tells the old indexes to delete after 10 days. All of these values can be customized to your specific needs.

    PUT _ilm/policy/apm_span_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "1d", 
            "max_size": "50gb" 
          }
        }
      },
      "delete": {
        "min_age": "10d", 
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

    Rollover after 1 day

    Rollover after 50gb

    Delete old indexes after 10 days

  3. Set up the ILM index template.

    To use the index lifecycle management policy created in the previous step, you need to specify it in the index template used to create the indices. The following template associates apm_span_policy with indices created from the apm-6.8.11-span-ilm template.

    Because we’re utilizing the current stack-version (6.8.11) in this step, this action will need to be performed as a part of each version upgrade.

    PUT _template/apm-6.8.11-span-ilm
{
  "order": 2,
  "index_patterns": ["apm-6.8.11-span-*"], 
  "settings": {
    "index.lifecycle.name": "apm_span_policy", 
    "index.lifecycle.rollover_alias": "apm-6.8.11-span"
  }
}

    This template applies to all indices with the prefix apm-6.8.11-span-

    Associates apm_span_policy with all indices created with this template

  4. Create the index and alias.

    Now we can create the first index: apm-6.8.11-span-000001.

    This action will need to be performed as a part of each version upgrade.

    PUT apm-6.8.11-span-000001 
{
  "aliases": {
    "apm-6.8.11-span":{
      "is_write_index": true 
    }
  }
}

    The rollover action increments the suffix number for each subsequent index.

    Designates this index as the write index for this alias.

  5. Verify the ILM index template was applied.

    GET apm-*-span/_settings

    The response should be similar to this:

    {
  "apm-6.8.11-span-000001" : {
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "apm_span_policy",
          "rollover_alias" : "apm-6.8.11-span"
        },
        "number_of_shards" : "1",
        "provided_name" : "apm-6.8.11-span-000001",
        "creation_date" : "1553024227938",
        "number_of_replicas" : "1",
        "uuid" : "6b5l-H7QTRK95FAodAN-wg",
        "version" : {
          "created" : "7000099"
        }
      }
    }
  }
}

    You can also verify and adjust your configuration via Kibana’s management.

  6. Repeat for other indices.

    Repeat the previous steps for each index that will be using ILM.

    • Create a policy
    • Set up the ILM index template
    • Create the index and alias
    • Verify the ILM index template was applied

  7. Modify APM Server’s configuration.

    Finally, modify the default index configuration in apm-server.yml. Trim off the date template from each index you are setting up ILM for, so that APM Server is always writing events to the same place. The name of your modified index configuration must match the is_write_index alias created previously

    It’s important to note that apm-server.yml overwrites defaults rather than being merged. This means you’ll need to configure all of your indices in the file, even if some are not using ILM.

    output.elasticsearch:
  indices:
    - index: "apm-6.8.11-sourcemap"
      when.contains:
        processor.event: "sourcemap"

    - index: "apm-6.8.11-error"
      when.contains:
        processor.event: "error"

    - index: "apm-6.8.11-transaction"
      when.contains:
        processor.event: "transaction"

    - index: "apm-6.8.11-span"
      when.contains:
        processor.event: "span"

    - index: "apm-6.8.11-metric"
      when.contains:
        processor.event: "metric"

    - index: "apm-6.8.11-onboarding"
      when.contains:
        processor.event: "onboarding"

  8. Start apm-server.

    Your ILM configuration should now be up and running!

    1. Monitor ILM status as events flow:

      GET apm-*/_ilm/explain?human

    2. Monitor index status:

      GET _cat/indices/apm*?v
« Manually loading template configuration Configure logging »