IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Date nanoseconds field type Flattened field type »
Elastic Docs ›Elasticsearch Guide [8.2] ›Mapping ›Field data types

Dense vector field type

edit

Dense vector field type

edit

The dense_vector field type stores dense vectors of float values. Dense vector fields can be used in the following ways:

  • In script_score queries, to score documents matching a filter
  • In the kNN search API, to find the k most similar vectors to a query vector

The dense_vector type does not support aggregations or sorting.

You add a dense_vector field as an array of floats:

PUT my-index
{
  "mappings": {
    "properties": {
      "my_vector": {
        "type": "dense_vector",
        "dims": 3
      },
      "my_text" : {
        "type" : "keyword"
      }
    }
  }
}

PUT my-index/_doc/1
{
  "my_text" : "text1",
  "my_vector" : [0.5, 10, 6]
}

PUT my-index/_doc/2
{
  "my_text" : "text2",
  "my_vector" : [-0.5, 10, 10]
}

Unlike most other data types, dense vectors are always single-valued. It is not possible to store multiple values in one dense_vector field.

Index vectors for kNN search

edit

This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.

A k-nearest neighbor (kNN) search finds the k nearest vectors to a query vector, as measured by a similarity metric.

Dense vector fields can be used to rank documents in script_score queries. This lets you perform a brute-force kNN search by scanning all documents and ranking them by similarity.

In many cases, a brute-force kNN search is not efficient enough. For this reason, the dense_vector type supports indexing vectors into a specialized data structure to support fast kNN retrieval through the kNN search API. You can enable indexing by setting the index parameter:

PUT my-index-2
{
  "mappings": {
    "properties": {
      "my_vector": {
        "type": "dense_vector",
        "dims": 3,
        "index": true,
        "similarity": "dot_product" 
      }
    }
  }
}

When index is enabled, you must define the vector similarity to use in kNN search

Elasticsearch uses the HNSW algorithm to support efficient kNN search. Like most kNN algorithms, HNSW is an approximate method that sacrifices result accuracy for improved speed.

Indexing vectors for approximate kNN search is an expensive process. It can take substantial time to ingest documents that contain vector fields with index enabled.

Dense vector fields cannot be indexed if they are within nested mappings.

Parameters for dense vector fields

edit

The following mapping parameters are accepted:

dims
(Required, integer) Number of vector dimensions. Can’t exceed 1024 for indexed vectors ("index": true), or 2048 for non-indexed vectors.
index
(Optional, Boolean) If true, you can search this field using the kNN search API. Defaults to false.
similarity

(Required*, string) The vector similarity metric to use in kNN search. Documents are ranked by their vector field’s similarity to the query vector. The _score of each document will be derived from the similarity, in a way that ensures scores are positive and that a larger score corresponds to a higher ranking.

* If index is true, this parameter is required.

Valid values for similarity
l2_norm
Computes similarity based on the L2 distance (also known as Euclidean distance) between the vectors. The document _score is computed as 1 / (1 + l2_norm(query, vector)^2).
dot_product
Computes the dot product of two vectors. This option provides an optimized way to perform cosine similarity. In order to use it, all vectors must be of unit length, including both document and query vectors. The document _score is computed as (1 + dot_product(query, vector)) / 2.
cosine
Computes the cosine similarity. Note that the most efficient way to perform cosine similarity is to normalize all vectors to unit length, and instead use dot_product. You should only use cosine if you need to preserve the original vectors and cannot normalize them in advance. The document _score is computed as (1 + cosine(query, vector)) / 2. The cosine similarity does not allow vectors with zero magnitude, since cosine is not defined in this case.

Although they are conceptually related, the similarity parameter is different from text field similarity and accepts a distinct set of options.

index_options

(Optional, object) An optional section that configures the kNN indexing algorithm. The HNSW algorithm has two internal parameters that influence how the data structure is built. These can be adjusted to improve the accuracy of results, at the expense of slower indexing speed. When index_options is provided, all of its properties must be defined.

Properties of index_options
type
(Required, string) The type of kNN algorithm to use. Currently only hnsw is supported.
m
(Required, integer) The number of neighbors each node will be connected to in the HNSW graph. Defaults to 16.
ef_construction
(Required, integer) The number of candidates to track while assembling the list of nearest neighbors for each new node. Defaults to 100.
« Date nanoseconds field type Flattened field type »