Tutorial: hybrid search with semantic_text

This tutorial demonstrates how to perform hybrid search, combining semantic search with traditional full-text search.

In hybrid search, semantic search retrieves results based on the meaning of the text, while full-text search focuses on exact word matches. By combining both methods, hybrid search delivers more relevant results, particularly in cases where relying on a single approach may not be sufficient.

The recommended way to use hybrid search in the Elastic Stack is following the semantic_text workflow. This tutorial uses the elasticsearch service for demonstration, but you can use any service and their supported models offered by the Inference API.

The destination index will contain both the embeddings for semantic search and the original text field for full-text search. This structure enables the combination of semantic search and full-text search.

PUT semantic-embeddings
{
  "mappings": {
    "properties": {
      "semantic_text": { 
        "type": "semantic_text",
      },
      "content": { 
        "type": "text",
        "copy_to": "semantic_text" 
      }
    }
  }
}

In this step, you load the data that you later use to create embeddings from.

Use the msmarco-passagetest2019-top1000 data set, which is a subset of the MS MARCO Passage Ranking data set. It consists of 200 queries, each accompanied by a list of relevant text passages. All unique passages, along with their IDs, have been extracted from that data set and compiled into a tsv file.

Download the file and upload it to your cluster using the Data Visualizer in the Machine Learning UI. After your data is analyzed, click Override settings. Under Edit field names, assign id to the first column and content to the second. Click Apply, then Import. Name the index test-data, and click Import. After the upload is complete, you will see an index named test-data with 182,469 documents.

Reindex the data from the test-data index into the semantic-embeddings index. The data in the content field of the source index is copied into the content field of the destination index. The copy_to parameter set in the index mapping creation ensures that the content is copied into the semantic_text field. The data is processed by the inference endpoint at ingest time to generate embeddings.

resp = client.reindex(
    wait_for_completion=False,
    source={
        "index": "test-data",
        "size": 10
    },
    dest={
        "index": "semantic-embeddings"
    },
)
print(resp)

const response = await client.reindex({
  wait_for_completion: "false",
  source: {
    index: "test-data",
    size: 10,
  },
  dest: {
    index: "semantic-embeddings",
  },
});
console.log(response);

POST _reindex?wait_for_completion=false
{
  "source": {
    "index": "test-data",
    "size": 10 
  },
  "dest": {
    "index": "semantic-embeddings"
  }
}

The call returns a task ID to monitor the progress:

resp = client.tasks.get(
    task_id="<task_id>",
)
print(resp)

const response = await client.tasks.get({
  task_id: "<task_id>",
});
console.log(response);

GET _tasks/<task_id>

Reindexing large datasets can take a long time. You can test this workflow using only a subset of the dataset.

To cancel the reindexing process and generate embeddings for the subset that was reindexed:

resp = client.tasks.cancel(
    task_id="<task_id>",
)
print(resp)

const response = await client.tasks.cancel({
  task_id: "<task_id>",
});
console.log(response);

POST _tasks/<task_id>/_cancel

After reindexing the data into the semantic-embeddings index, you can perform hybrid search by using reciprocal rank fusion (RRF). RRF is a technique that merges the rankings from both semantic and lexical queries, giving more weight to results that rank high in either search. This ensures that the final results are balanced and relevant.

resp = client.search(
    index="semantic-embeddings",
    retriever={
        "rrf": {
            "retrievers": [
                {
                    "standard": {
                        "query": {
                            "match": {
                                "content": "How to avoid muscle soreness while running?"
                            }
                        }
                    }
                },
                {
                    "standard": {
                        "query": {
                            "semantic": {
                                "field": "semantic_text",
                                "query": "How to avoid muscle soreness while running?"
                            }
                        }
                    }
                }
            ]
        }
    },
)
print(resp)

const response = await client.search({
  index: "semantic-embeddings",
  retriever: {
    rrf: {
      retrievers: [
        {
          standard: {
            query: {
              match: {
                content: "How to avoid muscle soreness while running?",
              },
            },
          },
        },
        {
          standard: {
            query: {
              semantic: {
                field: "semantic_text",
                query: "How to avoid muscle soreness while running?",
              },
            },
          },
        },
      ],
    },
  },
});
console.log(response);

GET semantic-embeddings/_search
{
  "retriever": {
    "rrf": {
      "retrievers": [
        {
          "standard": { 
            "query": {
              "match": {
                "content": "How to avoid muscle soreness while running?" 
              }
            }
          }
        },
        {
          "standard": { 
            "query": {
              "semantic": {
                "field": "semantic_text", 
                "query": "How to avoid muscle soreness while running?"
              }
            }
          }
        }
      ]
    }
  }
}

After performing the hybrid search, the query will return the top 10 documents that match both semantic and lexical search criteria. The results include detailed information about each document:

{
  "took": 107,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 473,
      "relation": "eq"
    },
    "max_score": null,
    "hits": [
      {
        "_index": "semantic-embeddings",
        "_id": "wv65epIBEMBRnhfTsOFM",
        "_score": 0.032786883,
        "_rank": 1,
        "_source": {
          "semantic_text": {
            "inference": {
              "inference_id": "my-elser-endpoint",
              "model_settings": {
                "task_type": "sparse_embedding"
              },
              "chunks": [
                {
                  "text": "What so many out there do not realize is the importance of what you do after you work out. You may have done the majority of the work, but how you treat your body in the minutes and hours after you exercise has a direct effect on muscle soreness, muscle strength and growth, and staying hydrated. Cool Down. After your last exercise, your workout is not over. The first thing you need to do is cool down. Even if running was all that you did, you still should do light cardio for a few minutes. This brings your heart rate down at a slow and steady pace, which helps you avoid feeling sick after a workout.",
                  "embeddings": {
                    "exercise": 1.571044,
                    "after": 1.3603843,
                    "sick": 1.3281639,
                    "cool": 1.3227621,
                    "muscle": 1.2645415,
                    "sore": 1.2561599,
                    "cooling": 1.2335974,
                    "running": 1.1750668,
                    "hours": 1.1104802,
                    "out": 1.0991782,
                    "##io": 1.0794281,
                    "last": 1.0474665,
                   (...)
                  }
                }
              ]
            }
          },
          "id": 8408852,
          "content": "What so many out there do not realize is the importance of (...)"
        }
      }
    ]
  }
}