Log retention

Enterprise Search indexes various event data such as analytics, API requests, and content source events. These event log indexes would grow infinitely in size without retention management. Since 7.9.0, Enterprise Search manages log retention for you, using Index Lifecycle Management (ILM).

Enterprise search creates several ILM policies that manage the Enterprise Search log indexes as they age, automatically transitioning each through a lifecycle. You can find these policies by their IDs to modify them, or you can create your own policies. See:

Enterprise Search uses ILM as long as the underlying Elastic Stack supports the feature. You can disable ILM on all self-managed deployments if desired. Also, App Search exposes additional controls to disable logging in that product only. See:

Enterprise Search creates ILM policies with the following IDs:

ent-search-api-ecs-ilm-logs
ent-search-app-search-analytics-ecs-ilm-logs
ent-search-workplace-search-analytics-ecs-ilm-logs
ent-search-workplace-search-content-events-ecs-ilm-logs

Manage the above Enterprise Search ILM policies using UIs in Kibana or APIs in Elasticsearch. See:

Enterprise Search automatically uses ILM for log retention management as long as the underlying Elastic Stack supports the feature.

However, if you’d prefer to manage log retention on your own, modify your Enterprise Search configuration. Set ilm.enabled to false.

From config/enterprise-search.yml:

# Use Index Lifecycle Management (ILM) to manage analytics and API logs
# retention.
#
# auto: Use ILM when supported by the underlying Elasticsearch cluster
# true: Use ILM (requires ILM support in the underlying Elasticsearch cluster)
# false: Don't use ILM (analytics and API logs will grow unconstrained)
#
#ilm.enabled: auto

Elastic Cloud deployments cannot access this setting, but such deployments can effectively disable ILM by modifying the ILM policies to discard logs. See Manage ILM policies, above.

Finally, App Search allows admins and developers to disable logging through its log settings UI and API. Although related to log retention, this is a separate feature that is specific to App Search.

See Log settings guide in the App Search documentation.