Merge

A shard in elasticsearch is a Lucene index, and a Lucene index is broken down into segments. Segments are internal storage elements in the index where the index data is stored, and are immutable up to delete markers. Segments are, periodically, merged into larger segments to keep the index size at bay and expunge deletes.

The more segments one has in the Lucene index means slower searches and more memory used. Segment merging is used to reduce the number of segments, however merges can be expensive to perform, especially on low IO environments. Merges can be throttled using store level throttling.

The index merge policy module allows one to control which segments of a shard index are to be merged. There are several types of policies with the default set to tiered.

Merges segments of approximately equal size, subject to an allowed number of segments per tier. This is similar to log_bytes_size merge policy, except this merge policy is able to merge non-adjacent segment, and separates how many segments are merged at once from how many segments are allowed per tier. This merge policy also does not over-merge (i.e., cascade merges).

This policy has the following settings:

For normal merging, this policy first computes a "budget" of how many segments are allowed to be in the index. If the index is over-budget, then the policy sorts segments by decreasing size (proportionally considering percent deletes), and then finds the least-cost merge. Merge cost is measured by a combination of the "skew" of the merge (size of largest seg divided by smallest seg), total merge size and pct deletes reclaimed, so that merges with lower skew, smaller size and those reclaiming more deletes, are favored.

If a merge will produce a segment that’s larger than max_merged_segment then the policy will merge fewer segments (down to 1 at once, if that one has deletions) to keep the segment size under budget.

Note, this can mean that for large shards that holds many gigabytes of data, the default of max_merged_segment (5gb) can cause for many segments to be in an index, and causing searches to be slower. Use the indices segments API to see the segments that an index has, and possibly either increase the max_merged_segment or issue an optimize call for the index (try and aim to issue it on a low traffic time).

A merge policy that merges segments into levels of exponentially increasing byte size, where each level has fewer segments than the value of the merge factor. Whenever extra segments (beyond the merge factor upper bound) are encountered, all segments within the level are merged.

This policy has the following settings:

A merge policy that tries to merge segments into levels of exponentially increasing document count, where each level has fewer segments than the value of the merge factor. Whenever extra segments (beyond the merge factor upper bound) are encountered, all segments within the level are merged.

The merge scheduler (ConcurrentMergeScheduler) controls the execution of merge operations once they are needed (according to the merge policy). Merges run in separate threads, and when the maximum number of threads is reached, further merges will wait until a merge thread becomes available. The merge scheduler supports this setting:

This is accepted for backwards compatibility, but just uses ConcurrentMergeScheduler with index.merge.scheduler.max_thread_count set to 1 so that only 1 merge may run at a time.