In this application the ingest of all the example documents is triggered with the 

 command. The implementation of this command is in the 

 function, which performs a complete import of all the documents stored in the 

The structure of each document is as follows:

: a URL to the document hosted on an external site

: a short summary of the contents of the document

: update date (could be missing if the document was never updated)

From these, this example application uses the 

 field as the text to be indexes, and adds 

The following snippet of Python code shows how the documents are imported:

 module from the Python standard library is used to read the data file, and then for each included document a 

 object from Langchain is created. Documents have a 

 attribute that defines the content to be converted into vectors and searched, plus a number of additional fields that are stored as metadata. The 

 determines which fields from the source content are to be stored as document metadata.

Depending on your ingest needs method can be refined or changed. The Langchain project provide a large selection of 

 that can be used depending on how the format of the source content.

The Elastic Learned Sparse EncodeR (ELSER) Model

The Elasticsearch index used in this application is configured to automatically create sparse vector embeddings for all documents that are inserted. The 

 makes sure that the ELSER model is installed and deployed on the Elasticsearch instance that you are using.

 field in these documents is long, which means that a single embedding will be unable to fully represent it. The standard solution when working with large amounts of text is to split the text into shorter 

, and then obtain embeddings for the individual passages, all of which are stored and indexed.

 encoder, which counts lengths of the passages in tokens, the same units that LLMs use.

Consider the following example, which demonstrates how text splitting works in the application:

 argument of the text splitter, it is possible to control the length of the resulting passages. The 

 allows for some amount of overlap between passages, which often helps obtain better embeddings.

In the actual application, the splitter is initialized with the following arguments:

You are welcome to change these values and see how changes affect the quality of the chatbot. Each time you change the splitter's configuration you should re-generate the index by running the 

Note that there are more considerations on 

text splitting in combination with the ELSER model

 that you should be aware of. For production use cases you might have to choose another tokenization method as in this tutorial.

Documents are stored in an Elasticsearch index. The name of the index is controlled by the 

 environment variable, which is defined in the 

 file. By default the name of this index is 

 class, which is part of the Elasticsearch integration in 

, and uses the official Elasticsearch client library for Python.

The complete logic that deals with the Elasticsearch index is shown below:

, writing them to the index given in the 

 argument. All operations are performed through the client given in the 

 argument defines the way this index is going to be used. For this application, the 

 class indicates that sparse vector embeddings are to be maintained for each document. This will add a 

 to the index that will generate embeddings through the requested model (ELSER version 2 in this case).

The Elasticsearch integration with Langchain provides a other strategies that can be used depending on the use case. In particular, the 

 can be used when dense vector embeddings are used.

Ingest

When you enter a question in the front end, a 

 endpoint is sent. The body of the request must include the question from the user, in the following format:

The response from the application is an event stream, as defined in the 

. The events that the server returns to the client have the following sequence:

data: [SESSION_ID] session-id-assigned-to-this-chat-session

 (repeated for each relevant document source that was identified)

 (repeated for each response chunk returned by the LLM)

The client can choose to ask a follow-up question by adding a 

 query string argument to the request URL.

The high-level logic for the chatbot endpoint is in the 

 function of the Flask application, in file 

 is a generator function that streams the events described above using Flask's response streaming feature, which is based on the 

Chatbot Endpoint

When a question is received, the application first searches the Elasticsearch index for relevant documents. This is achieved by generating a sparse vector embedding for the question, and then searching the index for the closest embeddings to it, each associated with a passage of a document.

As in the ingest phase, the Elasticsearch index is managed through the 

Generating an embedding for the question, and then searching for it is all nicely abstracted away by the 

 method of Langchain's retriever interface, which performs all of these tasks and returns the list of most relevant documents found:

You can see here how the returned passages are sent to the client as sources. The React application will show these as "Search Results" below the answer.

 class must match the strategy used during ingest. In this example, the 

 creates and searches sparse vectors from Elastic's ELSER model. Another interesting option you can evaluate is to use the 

Retrieval Phase

With the results from the search at hand, it is now possible to generate a prompt to send to the LLM. The prompt must include the original question sent by the user, the relevant passages that were obtained in the retrieval phase, and instructions to the LLM stating that the answer should come from the included passages.

To render the prompt, the application uses Flask's 

The template file that is referenced in this call is in 

You can make changes to this template if you'd like to see their effect in the quality of the chatbot responses. But always make sure to preserve the for-loop that renders the retrieved passages.

LLM Prompt

The prompt for the LLM is now ready, so what remains is to send it over and receive a response. To connect to LLMs, the application uses Langchain's streaming support, which nicely fits with the event streaming used in this application:

. Its purpose is to return the correct LLM integration from Langchain according to the configuration. Assuming you configured OpenAI, the LLM returned is going to be an instance of the 

Generation Phase

The process described above works well when users can only ask a single question. But this application allows follow-up questions as well, and this introduces a few additional complications. For example, there is a need to store all previous questions and answers, so that they can be included as additional context when sending the new question to the LLM.

The chat history in this application is managed through the 

 class, another class that is part of the Elasticsearch integration with Langchain. Each group of related questions and answers are written to an Elasticsearch index with a reference to the session ID that was used.

You may have noticed in the previous section that even though the response from the LLM is streamed out to the client in chunks, an 

 variable is generated with the full response. This is so that the response, along with its question, can be added to the history after each interaction:

 argument in the query string of the request URL, then the question is assumed to be made in the context of any previous questions under that same session.

The approach taken by this application for follow-up questions is to use the LLM to create a condensed question that summarizes the entire conversation, to be used for the retrieval phase. The purpose of this is to avoid running a vector search on a potentially large history of questions and answers. Here is the logic that performs this task:

This has a lot of similarities with how the main questions are handled, but in this case there is no need to use the streaming interface of the LLM, so the the 

To condense the question, a different prompt is used, stored in file **api/templates/condense_question_prompt.txt`:

This prompt renders all the questions and responses from the session, plus the new follow-up question at the end. The LLM is instructed to provide a simplified question that summarizes all the information.

To enable the LLM to have as much context as possible in the generation phase, the complete history of the conversation is added to the main prompt, along with the retrieved documents and the follow-up question. Here is the final version of the prompt as used in the example application:

You should note that the way the condensed question is used can be adapted to your needs. You may find that for some applications sending the condensed question also in the generation phase works better, also reducing the token count. Or perhaps not using a condensed question at all and always sending the entire chat history gives you better results. Hopefully now you have a good understanding of how this application works and can experiment with different prompts to find what works best for your use case.

Chat History

Chat History and Follow-Up Questions

This section of the tutorial discusses the most interesting aspects of the chatbot implementation, to help you understand it and customize it.

Implementation

Chatbot Implementation

Chatbots built with RAG can overcome some of the limitations that general-purpose conversational models such as ChatGPT have. In particular, they are able to discuss and answer questions about:

Information that is private to your organization.

Events that were not part of the training dataset, or that took place after the LLM finished training.

As an additional benefit, RAG helps to "ground" LLMs with facts, making them less likely to make up a response or "hallucinate".

The secret to achieve this is to use a two-step process to obtain an answer from the LLM:

, one or more data sources are searched for the user's query,. The relevant documents that are found in this search are retrieved. Using an 

 index for this is a great option for this, enabling you to choose between keyword, dense and sparse vector search methods, or even a hybrid combination of them.

, the user's prompt is expanded to include the documents retrieved in the first phase, with added instructions to the LLM to find the answer to the user's question in the retrieved information. The expanded prompt, including the added context for the question, is what is sent to the LLM in place of the original query.

This tutorial is structured in two main parts.

In the first part, you will learn how to run the 

 example, a complete application featuring a Python back end and a React front end.

Once you have the example application up and running, the second part of this tutorial explains the different components of the RAG implementation, to allow you to adapt the example code to your own needs.

Welcome

To follow this tutorial you will need to install the following components:

An installation of Elasticsearch, based on our hosted 

 service (which includes a free trial period), or a self-hosted service that you run on your own computer. See the 

 section above for installation instructions.

An API Key for OpenAI. You can actually use any other LLM that you favor, as long as it is supported by the 

 interpreter. Make sure it is a recent version, such as Python 3.8 or newer.

This tutorial focuses on RAG topics. To be able to modify the example application you will need basic knowledge of the following technologies:

Requirements

Project Setup

Running the Chatbot RAG Example

Python Back End

React Front End

Getting Started

You have reached the end of the Chatbot Tutorial. Congratulations!

We hope you now are familiar with the basic components of a chatbot project and the ideas behind 

You are encouraged to take our example application, experiment with it, and make it your own.

Chatbot Implementation

Ready to build state of the art search experiences?