Semantic search

[preview] This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.

Semantic search is a search method that helps you find data based on the intent and contextual meaning of a search query, instead of a match on query terms (lexical search).

Elasticsearch provides semantic search capabilities using natural language processing (NLP) and vector search. Deploying an NLP model to Elasticsearch enables it to extract text embeddings out of text. Embeddings are vectors that provide a numeric representation of a text. Pieces of content with similar meaning have similar representations.

A simplified representation of encoding textual concepts as vectors

At query time, Elasticsearch can use the same NLP model to convert a query into embeddings, enabling you to find documents with similar text embeddings.

This guide shows you how to implement semantic search with Elasticsearch, from selecting an NLP model, to writing queries.

Elasticsearch offers the usage of a wide range of NLP models, including both dense and sparse vector models. Your choice of the language model is critical for implementing semantic search successfully.

While it is possible to bring your own text embedding model, achieving good search results through model tuning is challenging. Selecting an appropriate model from our third-party model list is the first step. Training the model on your own data is essential to ensure better search results than using only BM25. However, the model training process requires a team of data scientists and ML experts, making it expensive and time-consuming.

To address this issue, Elastic provides a pre-trained representational model called Elastic Learned Sparse EncodeR (ELSER). ELSER, currently available only for English, is an out-of-domain sparse vector model that does not require fine-tuning. This adaptability makes it suitable for various NLP use cases out of the box. Unless you have a team of ML specialists, it is highly recommended to use the ELSER model.

In the case of sparse vector representation, the vectors mostly consist of zero values, with only a small subset containing non-zero values. This representation is commonly used for textual data. In the case of ELSER, each document in an index and the query text itself are represented by high-dimensional sparse vectors. Each non-zero element of the vector corresponds to a term in the model vocabulary. The ELSER vocabulary contains around 30000 terms, so the sparse vectors created by ELSER contain about 30000 values, the majority of which are zero. Effectively the ELSER model is replacing the terms in the original query with other terms that have been learnt to exist in the documents that best match the original search terms in a training dataset, and weights to control how important each is.

After you decide which model you want to use for implementing semantic search, you need to deploy the model in Elasticsearch.

Before you start using the deployed model to generate embeddings based on your input text, you need to prepare your index mapping first. The mapping of the index depends on the type of model.

curl -X PUT "${ES_URL}/my-index" \
-H "Authorization: ApiKey ${API_KEY}" \
-H "Content-Type: application/json" \
-d'
{
  "mappings": {
    "properties": {
      "my_tokens": {   
        "type": "sparse_vector"   
      },
      "my_text_field": {   
        "type": "text"   
      }
    }
  }
}
'

curl -X PUT "${ES_URL}/my-index" \
-H "Authorization: ApiKey ${API_KEY}" \
-H "Content-Type: application/json" \
-d'
{
  "mappings": {
    "properties": {
      "my_embeddings.predicted_value": {   
        "type": "dense_vector",   
        "dims": 384  
      },
      "my_text_field": {   
        "type": "text"   
      }
    }
  }
}
'

Once you have created the mappings for the index, you can generate text embeddings from your input text. This can be done by using an ingest pipeline with an inference processor. The ingest pipeline processes the input data and indexes it into the destination index. At index time, the inference ingest processor uses the trained model to infer against the data ingested through the pipeline. After you created the ingest pipeline with the inference processor, you can ingest your data through it to generate the model output.

curl -X PUT "${ES_URL}/_ingest/pipeline/my-text-embeddings-pipeline" \
-H "Authorization: ApiKey ${API_KEY}" \
-H "Content-Type: application/json" \
-d'
{
  "description": "Text embedding pipeline",
  "processors": [
    {
      "inference": {
        "model_id": ".elser_model_2",
        "input_output": [
          {
            "input_field": "my_text_field",
            "output_field": "my_tokens"
          }
        ]
      }
    }
  ]
}
'

curl -X PUT "${ES_URL}/_ingest/pipeline/my-text-embeddings-pipeline" \
-H "Authorization: ApiKey ${API_KEY}" \
-H "Content-Type: application/json" \
-d'
{
  "description": "Text embedding pipeline",
  "processors": [
    {
      "inference": {
        "model_id": "sentence-transformers__msmarco-minilm-l-12-v3",   
        "target_field": "my_embeddings",
        "field_map": {   
          "my_text_field": "text_field"
        }
      }
    }
  ]
}
'

Now it is time to perform semantic search!

Depending on the type of model you have deployed, you can query sparse vectors with a sparse vector query, or dense vectors with a kNN search.

curl -X GET "${ES_URL}/my-index/_search" \
-H "Authorization: ApiKey ${API_KEY}" \
-H "Content-Type: application/json" \
-d'
{
   "query":{
      "sparse_vector":{
         "field": "my_tokens",   
         "inference_id": "my-elser-endpoint",
         "query": "the query string"
         }
      }
   }
}
'

curl -X GET "${ES_URL}/my-index/_search" \
-H "Authorization: ApiKey ${API_KEY}" \
-H "Content-Type: application/json" \
-d'
{
  "knn": {
    "field": "my_embeddings.predicted_value",
    "k": 10,
    "num_candidates": 100,
    "query_vector_builder": {
      "text_embedding": {
        "model_id": "sentence-transformers__msmarco-minilm-l-12-v3",
        "model_text": "the query string"
      }
    }
  }
}
'

In some situations, lexical search may perform better than semantic search. For example, when searching for single words or IDs, like product numbers.

Combining semantic and lexical search into one hybrid search request using reciprocal rank fusion provides the best of both worlds. Not only that, but hybrid search using reciprocal rank fusion has been shown to perform better in general.

curl -X GET "${ES_URL}/my-index/_search" \
-H "Authorization: ApiKey ${API_KEY}" \
-H "Content-Type: application/json" \
-d'
{
  "retriever": {
    "rrf": {
      "retrievers": [
        {
          "standard": {
            "query": {
              "match": {
                "my_text_field": "the query string"
              }
            }
          }
        },
        {
          "standard": {
            "query": {
              "sparse_vector": {
                "field": "my_tokens",
                "inference_id": "my-elser-endpoint",
                "query": "the query string"
              }
            }
          }
        }
      ],
      "window_size": 50,
      "rank_constant": 20
    }
  }
}
'

curl -X GET "${ES_URL}/my-index/_search" \
-H "Authorization: ApiKey ${API_KEY}" \
-H "Content-Type: application/json" \
-d'
{
  "query": {
    "match": {
      "my_text_field": "the query string"
    }
  },
  "knn": {
    "field": "text_embedding.predicted_value",
    "k": 10,
    "num_candidates": 100,
    "query_vector_builder": {
      "text_embedding": {
        "model_id": "sentence-transformers__msmarco-minilm-l-12-v3",
        "model_text": "the query string"
      }
    }
  },
  "rank": {
    "rrf": {}
  }
}
'