Retrievers
editRetrievers
editA retriever is an abstraction that was added to the Search API in 8.14.0 and was made generally available in 8.16.0.
This abstraction enables the configuration of multi-stage retrieval pipelines within a single _search
call.
This simplifies your search application logic, because you no longer need to configure complex searches via multiple Elasticsearch calls or implement additional client-side logic to combine results from different queries.
This document provides a general overview of the retriever abstraction.
For implementation details, including notable restrictions, check out the
reference documentation in the _search
API docs.
Retriever types
editRetrievers come in various types, each tailored for different search operations. The following retrievers are currently available:
-
Standard Retriever. Returns top documents from a
traditional query.
Mimics a traditional query but in the context of a retriever framework. This
ensures backward compatibility as existing
_search
requests remain supported. That way you can transition to the new abstraction at your own pace without mixing syntaxes. - kNN Retriever. Returns top documents from a knn search, in the context of a retriever framework.
-
RRF Retriever. Combines and ranks multiple first-stage retrievers using
the reciprocal rank fusion (RRF) algorithm. Allows you to combine multiple result sets
with different relevance indicators into a single result set.
An RRF retriever is a compound retriever, where its
filter
element is propagated to its sub retrievers. -
Text Similarity Re-ranker Retriever. Used for semantic reranking.
Requires first creating a
rerank
task using the Elasticsearch Inference API.
What makes retrievers useful?
editHere’s an overview of what makes retrievers useful and how they differ from regular queries.
- Simplified user experience. Retrievers simplify the user experience by allowing entire retrieval pipelines to be configured in a single API call. This maintains backward compatibility with traditional query elements by automatically translating them to the appropriate retriever.
- Structured retrieval. Retrievers provide a more structured way to define search operations. They allow searches to be described using a "retriever tree", a hierarchical structure that clarifies the sequence and logic of operations, making complex searches more understandable and manageable.
- Composability and flexibility. Retrievers enable flexible composability, allowing you to build pipelines and seamlessly integrate different retrieval strategies into these pipelines. Retrievers make it easy to test out different retrieval strategy combinations.
- Compound operations. A retriever can have sub retrievers. This allows complex nested searches where the results of one retriever feed into another, supporting sophisticated querying strategies that might involve multiple stages or criteria.
- Retrieval as a first-class concept. Unlike traditional queries, where the query is a part of a larger search API call, retrievers are designed as standalone entities that can be combined or used in isolation. This enables a more modular and flexible approach to constructing searches.
-
Enhanced control over document scoring and ranking.
Retrievers allow for more explicit control over how documents are scored and filtered.
For instance, you can specify minimum score thresholds, apply complex filters without affecting scoring, and use parameters like
terminate_after
for performance optimizations. -
Integration with existing Elasticsearch functionalities.
Even though retrievers can be used instead of existing
_search
API syntax (like thequery
andknn
), they are designed to integrate seamlessly with things like pagination (search_after
) and sorting. They also maintain compatibility with aggregation operations by treating the combination of all leaf retrievers asshould
clauses in a boolean query. - Cleaner separation of concerns. When using compound retrievers, only the query element is allowed, which enforces a cleaner separation of concerns and prevents the complexity that might arise from overly nested or interdependent configurations.
Example
editThe following example demonstrates the powerful queries that we can now compose, and how retrievers simplify this process. We can use any combination of retrievers we want, propagating the
results of a nested retriever to its parent. In this scenario, we’ll make use of all 4 (currently) available retrievers, i.e. standard
, knn
, text_similarity_reranker
and rrf
.
We’ll first combine the results of a semantic
query using the standard
retriever, and that of a knn
search on a dense vector field, using rrf
to get the top 100 results.
Finally, we’ll then rerank the top-50 results of rrf
using the text_similarity_reranker
GET example-index/_search { "retriever": { "text_similarity_reranker": { "retriever": { "rrf": { "retrievers": [ { "standard": { "query": { "semantic": { "field": "inference_field", "query": "state of the art vector database" } } } }, { "knn": { "query_vector": [ 0.54, ..., 0.245 ], "field": "embedding", "k": 10, "num_candidates": 15 } } ], "rank_window_size": 100, "rank_constant": 10 } }, "rank_window_size": 50, "field": "description", "inference_text": "what's the best way to create complex pipelines and retrieve documents?", "inference_id": "my-awesome-rerank-model" } } }
Glossary
editHere are some important terms:
- Retrieval Pipeline. Defines the entire retrieval and ranking logic to produce top hits.
- Retriever Tree. A hierarchical structure that defines how retrievers interact.
- First-stage Retriever. Returns an initial set of candidate documents.
- Compound Retriever. Builds on one or more retrievers, enhancing document retrieval and ranking logic.
- Combiners. Compound retrievers that merge top hits from multiple sub-retrievers.
- Rerankers. Special compound retrievers that reorder hits and may adjust the number of hits, with distinctions between first-stage and second-stage rerankers.
Retrievers in action
editThe Search Playground builds Elasticsearch queries using the retriever abstraction. It automatically detects the fields and types in your index and builds a retriever tree based on your selections.
You can use the Playground to experiment with different retriever configurations and see how they affect search results.
Refer to the Playground documentation for more information.
API reference
editFor implementation details, including notable restrictions, check out the reference documentation in the Search API docs.