History retention

Elasticsearch sometimes needs to replay some of the operations that were performed on a shard. For instance, if a replica is briefly offline then it may be much more efficient to replay the few operations it missed while it was offline than to rebuild it from scratch. Similarly, cross-cluster replication works by performing operations on the leader cluster and then replaying those operations on the follower cluster.

At the Lucene level there are really only two write operations that Elasticsearch performs on an index: a new document may be indexed, or an existing document may be deleted. Updates are implemented by atomically deleting the old document and then indexing the new document. A document indexed into Lucene already contains all the information needed to replay that indexing operation, but this is not true of document deletions. To solve this, Elasticsearch uses a feature called soft deletes to preserve recent deletions in the Lucene index so that they can be replayed.

Elasticsearch only preserves certain recently-deleted documents in the index because a soft-deleted document still takes up some space. Eventually Elasticsearch will fully discard these soft-deleted documents to free up that space so that the index does not grow larger and larger over time. Fortunately Elasticsearch does not need to be able to replay every operation that has ever been performed on a shard, because it is always possible to make a full copy of a shard on a remote node. However, copying the whole shard may take much longer than replaying a few missing operations, so Elasticsearch tries to retain all of the operations it expects to need to replay in future.

Elasticsearch keeps track of the operations it expects to need to replay in future using a mechanism called shard history retention leases. Each shard copy that might need operations to be replayed must first create a shard history retention lease for itself. For example, this shard copy might be a replica of a shard or it might be a shard of a follower index when using cross-cluster replication. Each retention lease keeps track of the sequence number of the first operation that the corresponding shard copy has not received. As the shard copy receives new operations, it increases the sequence number contained in its retention lease to indicate that it will not need to replay those operations in future. Elasticsearch discards soft-deleted operations once they are not being held by any retention lease.

If a shard copy fails then it stops updating its shard history retention lease, which means that Elasticsearch will preserve all new operations so they can be replayed when the failed shard copy recovers. However, retention leases only last for a limited amount of time. If the shard copy does not recover quickly enough then its retention lease may expire. This protects Elasticsearch from retaining history forever if a shard copy fails permanently, because once a retention lease has expired Elasticsearch can start to discard history again. If a shard copy recovers after its retention lease has expired then Elasticsearch will fall back to copying the whole index since it can no longer simply replay the missing history. The expiry time of a retention lease defaults to 12h which should be long enough for most reasonable recovery scenarios.