Index lifecycle management (ILM)edit
The index lifecycle management (ILM) feature in Elasticsearch allows you to automate the lifecycle of your APM Server indices as they grow and age.
ILM is enabled by default in v7.3+, and the default ILM policy will be applied to all APM indices as long as all of the following conditions are true:
-
output.elasticsearch
is enabled. - The Elasticsearch instance supports ILM.
-
Custom
index
orindices
settings are not configured.
If you wish to define your own index lifecycle policies, see Custom configuration.
ILM policies are only applied to indices created after such policies have taken effect. This means any indices present before ILM was configured will need to be managed manually.
Default policyedit
ILM manages an index based on its defined policy.
The default ILM configuration applies hot and warm phases to all APM events:
span
, transaction
, error
, and metric
.
Cold and delete phases are not defined.
-
Hot — Rollover data when the index reaches a maximum size of 50gb or a maximum age of 30 days:
max_size: 50gb
,max_age: 30d
-
Warm — Move to warm phase after 30 days:
min_age: 30d
Custom configurationedit
If you’d like APM Server to assist you in creating a custom index lifecycle policy,
you must first ensure that ILM and ILM setup are enabled.
You can then create a custom policy and map it to the correct APM event types.
All of this can be done directly from the apm-server.yml
file.
The following table can help you determine which options to configure when enabling, setting up, or disabling ILM:
Enable ILM && Enable ILM setup. |
|
Details: Create managed indices. Default policies and templates are written to Elasticsearch. Successful if: A policy and template do not exist. Example: Your first time starting APM Server |
Configure these options: |
Details: Create managed indices. Customized policies and templates are written to Elasticsearch. Successful if: A policy and template already exists. Example: You’re customizing a policy that’s already been sent to Elasticsearch. |
Configure these options: |
Disable ILM && Enable ILM setup. |
|
Details: Create unmanaged indices. Ensure event specific templates are set up accordingly. |
Configure these options: |
Enable ILM && Disable ILM setup. |
|
Details: Use managed indices. No policies and templates will be created by APM Server. Example: You’re creating your policies and templates elsewhere, e.g., with Elasticsearch APIs. |
Configure these options: |
Using custom Indices |
|
Details: Configure event specific templates when using custom index names. |
Configure these options: |
Enable ILMedit
ILM is enabled/disabled with apm-server.ilm.enabled
:
apm-server: ilm: enabled: "auto"
ilm.enabled
edit
Can be "auto"
, "true"
, or "false"
. Defaults to "auto"
in v7.3+.
-
"auto"
APM Server will enable ILM only if all three of the conditions below are true. If any of the conditions are not met, ILM will be disabled.
-
output.elasticsearch
is enabled. - The Elasticsearch instance supports ILM.
-
Custom
index
orindices
settings are not configured.
-
-
"true"
APM Server will ignore any configured index settings and enable ILM if both of the conditions below are true. If any of the conditions are not met, APM Server will disable ILM, create unmanaged indices, and an error will be logged.
-
output.elasticsearch
is enabled. - The Elasticsearch instance supports ILM.
-
-
"false"
Explicitly disables ILM.
Set up ILMedit
ILM setup can be customized with the apm-server.ilm.setup
configuration options.
The default configuration enables APM Server to handle all of the setup required for ILM:
apm-server: ilm: setup: enabled: true overwrite: true require_policy: true mapping: - event_type: "error" policy_name: "apm-rollover-30-days" - event_type: "span" policy_name: "apm-rollover-30-days" - event_type: "transaction" policy_name: "apm-rollover-30-days" - event_type: "metric" policy_name: "apm-rollover-30-days"
apm-server.ilm.setup.enabled
edit
Defaults to true
.
When true
, APM Server will create an ILM specific index template for each APM event type.
This is required to map ILM aliases and policies to indices.
When false
, ILM setup is disabled. No policies, templates, or aliases will be created by APM Server.
Only disable ilm.setup
if you want to set up index management on your own.
If you simply want to disable ILM, use apm-server.ilm.enabled: false
instead.
apm-server.ilm.setup.overwrite
edit
Defaults to false
. When false
, APM Server will not overwrite any existing policies or ILM related templates.
When first setting up ILM, your initial template and policy will be applied.
You must set this to true
when customizing your policies and template for them to be applied,
or if you want to switch between managed and unmanaged indices.
apm-server.ilm.setup.require_policy
edit
Defaults to true
, which means that an ILM policy must be defined in apm-server.yml
.
Changing this to false
allows you to manually set up ILM policies and templates outside of APM Server,
e.g., with Elasticsearch APIs.
APM Server will still make use of ILM and connect your template with the defined mapping.
apm-server.ilm.setup.mapping
edit
Maps each event type to the named policy. APM event types can only be error
, span
, transaction
, and metric
.
Policies defined must be mapped to an event type. If they are not, they will not be sent to Elasticsearch.
If you attempt to map an index lifecycle policy to a different event type, APM Server will not start.
If you only map a subset of APM event types, the default values will be used for omitted event types.
Create a custom ILM policyedit
Policies only need to be created once and will persist through version upgrades.
Any change in existing ILM policies will only take place once the next phase is entered.
You can define as many policies as you’d like.
Just make sure to include the policy name in the ilm.setup.mapping
.
If your policy isn’t mapped to an event type, it will not be sent to Elasticsearch.
APM Server doesn’t do any validation on policies.
Instead, if something is incorrectly defined, Elasticsearch will respond with 400
and APM Server won’t connect.
The default ILM policy can be viewed and edited in two places:
-
In your
apm-server.yml
configuration file. - On the Index lifecycle policies page in Kibana.
Head on over to the Elasticsearch documentation to learn more about all available policy phases and actions.
After starting up APM Server, you can confirm the policy was created by using the GET lifecycle policy API:
GET _ilm/policy
Example ILM configurationedit
Here’s what a custom ILM configuration might look like.
The example below creates two different policies, one for errors
and spans
,
and another for transactions
and metrics
.
The apm-error-span-policy
applies all four phases to its index lifecycle, including a cold phase with frozen indices,
and a delete phase after 30 days.
The apm-transaction-metric-policy
keeps data in the hot, warm, and cold phases for a longer period of time,
and does not delete any data.
ilm: enabled: "auto" setup: mapping: - event_type: "error" policy_name: "apm-error-span-policy" - event_type: "span" policy_name: "apm-error-span-policy" - event_type: "transaction" policy_name: "apm-transaction-metric-policy" - event_type: "metric" policy_name: "apm-transaction-metric-policy" enabled: true policies: - name: "apm-error-span-policy" policy: phases: hot: actions: rollover: max_size: "50gb" max_age: "1d" set_priority: priority: 100 warm: min_age: "7d" actions: set_priority: priority: 50 readonly: {} cold: min_age: "30d" actions: set_priority: priority: 0 freeze: {} delete: min_age: "60d" actions: delete: {} - name: "apm-transaction-metric-policy" policy: phases: hot: actions: rollover: max_size: "50gb" max_age: "30d" set_priority: priority: 100 warm: min_age: "60d" actions: set_priority: priority: 50 readonly: {} cold: min_age: "90d" actions: set_priority: priority: 0 freeze: {}