In the previous chapter you have seen how to expand an Elasticsearch index with a 

 field that is populated with embeddings generated by a Machine Learning model. The model was installed locally on your computer, and the embeddings were generated from the Python code and added to the documents before they were inserted into the index.

In this chapter you are going to learn about another vector type, the 

, which is designed to store inferences from the 

Elastic Learned Sparse EncodeR model (ELSER)

. Embeddings returned by this model are a collection of tags (more appropriately called 

In this chapter you will also use a different method for working with Machine Learning models, in which the Elasticsearch service itself runs the model and adds the resulting embeddings to the index through a 

 field type you used in the previous chapter, the 

 type can store inferences returned by Machine Learning models. While dense vectors hold a fixed-length array of numbers that describe the source text, a sparse vector stores a mapping of features to weights.

 field to the index. This is a type that needs to be defined explicitly in the index mapping. Below you can see an updated version of the 

As mentioned above, in this example Elasticsearch will take ownership of the model and automatically execute it to generate embeddings, both when inserting documents and when searching.

The Elasticsearch client exposes a set of API endpoints to manage Machine Learning models and their pipelines. The following 

 follows a few steps to download and install the ELSER v2 model, and to create a pipeline that uses it to populate the 

Configuring ELSER for us requires a several steps. First, the 

 method of the Elasticsearch is used to download ELSER. The 

 argument identifies the model and version to download (ELSER v2 is available for Elasticsearch 8.11 and up). The 

 field is the configuration required by this model.

Once the model is downloaded it needs to be deployed. For this, the 

 method is used, just with the identifier of the model to deploy. Note that this is an asynchronous operation, so the model is going to be available for use after a short amount of time.

The final step to configure the use of ELSER is to define a pipeline for it. A pipeline is used to tell Elasticsearch how the model has to be used. A pipeline is given an identifier and one or more processing tasks to perform. The pipeline created above is called 

 and has a single inference task, which means that each time a document is added, the model is going to run with on the 

, and the output will be added to the document on the 

 field is used to generate the embeddings, as with the dense vector embeddings in the previous chapter. The resulting embeddings are going to be written to the 

 sparse vector field created in the previous section.

To make it easy to invoke this method, add a 

You can now deploy ELSER on your Elasticsearch service with the following command:

The last configuration task involves linking the index with the pipeline, so that the model is automatically executed when documents are inserted on this index. This is done on the index configuration with a 

With this change, you can now regenerate the index with full support for ELSER inferences:

ELSER Model

The ELSER Model

With the index now equipped with ELSER embeddings, the 

 can be changed to search these embeddings. For now, you'll see how to search only through ELSER, later the previous search methods will be incorporated back to create a combined solution.

To use ELSER inferences when searching, the 

 query type is used. Below you can see an updated 

 query receives a key with the name of the field to be searched. Under this key, 

 configures which model to use in the search, and 

 defines what to search for. Note how in this case there is no need to generate an embedding for the search text, as Elasticsearch manages the model and can take care of that.

 have been left unused, and the aggregations have been omitted. These can be added back in the same way they were incorporated into the full-text search solution. Below is an updated 

Spend some time experimenting with different searches. You will notice that as with dense vector embeddings, searches driven by the ELSER model work better than full-text search when the exact words do not appear in the indexed documents.

Semantic Queries

The solution to implementing a hybrid full-text and dense vector search was to send a search request that included the 

 arguments to request the two searches, and the 

 argument to combine them into a single results list.

The complication that is presented when trying to do the same to combine full-text and sparse vector search requests is that both use the 

 argument. To be able to provide the two queries that need to be combined with the RRF algorithm, it is necessary to include two 

 arguments, and the solution to do this is to do it with 

Sub-searches is a feature that is currently in technical preview. For this reason the Python Elasticsearch client does not natively support it. To work around this limitation, the 

 class can be changed to send the search request using the 

 argument. Below you can see a new, yet similar implementation that uses the 

 argument of the client to send a search request:

This implementation does not require any changes to the application, as it is functionally equivalent. The only difference is that the 

 method validates all arguments before sending the request, with 

 being the only exception. The server always validates requests regardless of how the client sends them.

 to send multiple search queries as follows:

To complete this section, let's bring back the full-text logic and combine it with the semantic search query presented earlier in this chapter.

 function looked for category filters entered by the user on the search prompt, and returned the left over portion as 

 is empty, it means that the user only enter a category filter, and in that case the query should be a simple 

 with the selected category as a filter. This is implemented in the 

 option is used as shown in the previous section to include the 

 option requesting that the results from the two sub-searches are combined into a single list of ranked results. To complete the query, the 

 argument are provided to maintain the support for pagination.

 to review this version of the application.

Hybrid Search

Hybrid Search: Combined Full-Text and ELSER Results

In this section you are going to learn about another Machine Learning approach to search that utilizes the 

Elastic Learned Sparse EncodeR model, or ELSER

, a Natural Language Processing model trained by Elastic to perform semantic search.

Semantic Search

Semantic Search using ELSER

This hands-on tutorial will teach you how to build a complete search solution using 

How to perform full-text keyword searches on a dataset, optionally with filters

How to generate, store and search dense vector embeddings using a Machine Learning model

 model to generate and search sparse vectors

How to combine search results from the methods listed above using Elastic's 

This most important aspect of this tutorial is that it will show you how to implement all these features on a project that you will run on your own computer, all done in small incremental steps.

The examples you will learn are written in 

, but the concepts are universal and can be applied to your favorite language or technology stack.

To get the most out of this tutorial, we recommend that you follow along and run all the examples.

Welcome

To follow this tutorial you will need to install the following components:

An installation of Elasticsearch, based on our hosted 

 service (which includes a free trial period), or a self-hosted service that you run on your own computer. See the 

 section above for installation instructions.

 interpreter. Make sure it is a recent version, such as Python 3.8 or newer.

The tutorial assumes that you have no previous knowledge of Elasticsearch or general search topics, but it expects you to have a basic familiarity with the following technologies, at least at a beginner level:

The command prompt or terminal application in your operating system.

Requirements

Download the starter search application by clicking on the link below.

Download the Tutorial Starter Application

Find a suitable parent directory for your project, such as your 

 directory, and extract the contents of the zip file there. This should add a 

 directory with several sub-directories and files inside.

 directory created in the previous section.

Following Python best practices, you are now going to create a 

, a private Python environment dedicated to this project. Do this with the following command:

This command creates a Python virtual environment in a 

 in this command with any name that you like. Note that in some installations of Python, you may need to use 

 the virtual environment, which is a way to make this virtual environment the active Python environment for the terminal session you are in. If you are working on a UNIX-based operating system such as Linux or macOS, activate the virtual environment as follows:

The above activation command would also work if you are working inside a WSL environment on a Microsoft Windows computer. But if you are using the Windows command prompt or PowerShell, the activation command is different:

When the virtual environment is activated, the command-line prompt changes to show the name of the environment:

The last step to configure the Python environment is to install a few packages that are needed by the starter application. Make sure that the virtual environment was activated in the previous step, and then run the following command to install these dependencies:

At this point you should be able to start the application with the following command:

To confirm that the application is running, open your browser and navigate to 

The Flask application is configured to run in development mode. When it detects that a source file has been changed, it will restart itself automatically to incorporate the changes. You can leave this terminal session with the application running while you continue with the tutorial, and as you make changes the application will restart to update.

Project Setup

Python Client Setup

Set Up the Elasticsearch Python Client

Create an Index

Create an Elasticsearch Index

Search Basics

Pagination

Filters

Faceted Search

In this section of the tutorial you are going to learn how to work with the full-text search capabilities of Elasticsearch. This will introduce you to several core topics such as working with the Python client library, creating and updating indices, making queries, and working with search results, all of which also apply to the more advanced search methods you will learn later.

Full-Text Search

Embeddings Intro

Introduction to Embeddings

Generating Embeddings

Generate Embeddings

Storing Embeddings

Store Embeddings in Elasticsearch

k-Nearest Neighbor Search

k-Nearest Neighbor (kNN) Search

Hybrid Search: Combined Full-Text and kNN Results

This section will introduce you to a different way of searching that leverages Machine Learning (ML) techniques to interpret meaning and context.

Vector Search

Congratulations, you have finished the Search Tutorial!

We hope that this tutorial has given you a base from which you can start to experiment and create your search solution with Elasticsearch!

Semantic Search using ELSER

Ready to build state of the art search experiences?