- Elasticsearch Guide: other versions:
- What’s new in 8.17
- Elasticsearch basics
- Quick starts
- Set up Elasticsearch
- Run Elasticsearch locally
- Installing Elasticsearch
- Configuring Elasticsearch
- Important Elasticsearch configuration
- Secure settings
- Auditing settings
- Circuit breaker settings
- Cluster-level shard allocation and routing settings
- Miscellaneous cluster settings
- Cross-cluster replication settings
- Discovery and cluster formation settings
- Data stream lifecycle settings
- Field data cache settings
- Local gateway settings
- Health Diagnostic settings
- Index lifecycle management settings
- Index management settings
- Index recovery settings
- Indexing buffer settings
- Inference settings
- License settings
- Machine learning settings
- Monitoring settings
- Node settings
- Networking
- Node query cache settings
- Path settings
- Search settings
- Security settings
- Shard request cache settings
- Snapshot and restore settings
- Transforms settings
- Thread pools
- Watcher settings
- Set JVM options
- Important system configuration
- Bootstrap Checks
- Heap size check
- File descriptor check
- Memory lock check
- Maximum number of threads check
- Max file size check
- Maximum size virtual memory check
- Maximum map count check
- Client JVM check
- Use serial collector check
- System call filter check
- OnError and OnOutOfMemoryError checks
- Early-access check
- All permission check
- Discovery configuration check
- Bootstrap Checks for X-Pack
- Starting Elasticsearch
- Stopping Elasticsearch
- Discovery and cluster formation
- Add and remove nodes in your cluster
- Full-cluster restart and rolling restart
- Remote clusters
- Plugins
- Upgrade Elasticsearch
- Index modules
- Mapping
- Dynamic mapping
- Explicit mapping
- Runtime fields
- Field data types
- Aggregate metric
- Alias
- Arrays
- Binary
- Boolean
- Completion
- Date
- Date nanoseconds
- Dense vector
- Flattened
- Geopoint
- Geoshape
- Histogram
- IP
- Join
- Keyword
- Nested
- Numeric
- Object
- Pass-through object
- Percolator
- Point
- Range
- Rank feature
- Rank features
- Search-as-you-type
- Semantic text
- Shape
- Sparse vector
- Text
- Token count
- Unsigned long
- Version
- Metadata fields
- Mapping parameters
analyzer
coerce
copy_to
doc_values
dynamic
eager_global_ordinals
enabled
format
ignore_above
index.mapping.ignore_above
ignore_malformed
index
index_options
index_phrases
index_prefixes
meta
fields
normalizer
norms
null_value
position_increment_gap
properties
search_analyzer
similarity
store
subobjects
term_vector
- Mapping limit settings
- Removal of mapping types
- Text analysis
- Overview
- Concepts
- Configure text analysis
- Built-in analyzer reference
- Tokenizer reference
- Token filter reference
- Apostrophe
- ASCII folding
- CJK bigram
- CJK width
- Classic
- Common grams
- Conditional
- Decimal digit
- Delimited payload
- Dictionary decompounder
- Edge n-gram
- Elision
- Fingerprint
- Flatten graph
- Hunspell
- Hyphenation decompounder
- Keep types
- Keep words
- Keyword marker
- Keyword repeat
- KStem
- Length
- Limit token count
- Lowercase
- MinHash
- Multiplexer
- N-gram
- Normalization
- Pattern capture
- Pattern replace
- Phonetic
- Porter stem
- Predicate script
- Remove duplicates
- Reverse
- Shingle
- Snowball
- Stemmer
- Stemmer override
- Stop
- Synonym
- Synonym graph
- Trim
- Truncate
- Unique
- Uppercase
- Word delimiter
- Word delimiter graph
- Character filters reference
- Normalizers
- Index templates
- Data streams
- Ingest pipelines
- Example: Parse logs
- Enrich your data
- Processor reference
- Append
- Attachment
- Bytes
- Circle
- Community ID
- Convert
- CSV
- Date
- Date index name
- Dissect
- Dot expander
- Drop
- Enrich
- Fail
- Fingerprint
- Foreach
- Geo-grid
- GeoIP
- Grok
- Gsub
- HTML strip
- Inference
- IP Location
- Join
- JSON
- KV
- Lowercase
- Network direction
- Pipeline
- Redact
- Registered domain
- Remove
- Rename
- Reroute
- Script
- Set
- Set security user
- Sort
- Split
- Terminate
- Trim
- Uppercase
- URL decode
- URI parts
- User agent
- Ingest pipelines in Search
- Aliases
- Search your data
- Re-ranking
- Query DSL
- Aggregations
- Bucket aggregations
- Adjacency matrix
- Auto-interval date histogram
- Categorize text
- Children
- Composite
- Date histogram
- Date range
- Diversified sampler
- Filter
- Filters
- Frequent item sets
- Geo-distance
- Geohash grid
- Geohex grid
- Geotile grid
- Global
- Histogram
- IP prefix
- IP range
- Missing
- Multi Terms
- Nested
- Parent
- Random sampler
- Range
- Rare terms
- Reverse nested
- Sampler
- Significant terms
- Significant text
- Terms
- Time series
- Variable width histogram
- Subtleties of bucketing range fields
- Metrics aggregations
- Pipeline aggregations
- Average bucket
- Bucket script
- Bucket count K-S test
- Bucket correlation
- Bucket selector
- Bucket sort
- Change point
- Cumulative cardinality
- Cumulative sum
- Derivative
- Extended stats bucket
- Inference bucket
- Max bucket
- Min bucket
- Moving function
- Moving percentiles
- Normalize
- Percentiles bucket
- Serial differencing
- Stats bucket
- Sum bucket
- Bucket aggregations
- Geospatial analysis
- Connectors
- EQL
- ES|QL
- SQL
- Overview
- Getting Started with SQL
- Conventions and Terminology
- Security
- SQL REST API
- SQL Translate API
- SQL CLI
- SQL JDBC
- SQL ODBC
- SQL Client Applications
- SQL Language
- Functions and Operators
- Comparison Operators
- Logical Operators
- Math Operators
- Cast Operators
- LIKE and RLIKE Operators
- Aggregate Functions
- Grouping Functions
- Date/Time and Interval Functions and Operators
- Full-Text Search Functions
- Mathematical Functions
- String Functions
- Type Conversion Functions
- Geo Functions
- Conditional Functions And Expressions
- System Functions
- Reserved keywords
- SQL Limitations
- Scripting
- Data management
- ILM: Manage the index lifecycle
- Tutorial: Customize built-in policies
- Tutorial: Automate rollover
- Index management in Kibana
- Overview
- Concepts
- Index lifecycle actions
- Configure a lifecycle policy
- Migrate index allocation filters to node roles
- Troubleshooting index lifecycle management errors
- Start and stop index lifecycle management
- Manage existing indices
- Skip rollover
- Restore a managed data stream or index
- Data tiers
- Autoscaling
- Monitor a cluster
- Roll up or transform your data
- Set up a cluster for high availability
- Snapshot and restore
- Secure the Elastic Stack
- Elasticsearch security principles
- Start the Elastic Stack with security enabled automatically
- Manually configure security
- Updating node security certificates
- User authentication
- Built-in users
- Service accounts
- Internal users
- Token-based authentication services
- User profiles
- Realms
- Realm chains
- Security domains
- Active Directory user authentication
- File-based user authentication
- LDAP user authentication
- Native user authentication
- OpenID Connect authentication
- PKI user authentication
- SAML authentication
- Kerberos authentication
- JWT authentication
- Integrating with other authentication systems
- Enabling anonymous access
- Looking up users without authentication
- Controlling the user cache
- Configuring SAML single-sign-on on the Elastic Stack
- Configuring single sign-on to the Elastic Stack using OpenID Connect
- User authorization
- Built-in roles
- Defining roles
- Role restriction
- Security privileges
- Document level security
- Field level security
- Granting privileges for data streams and aliases
- Mapping users and groups to roles
- Setting up field and document level security
- Submitting requests on behalf of other users
- Configuring authorization delegation
- Customizing roles and authorization
- Enable audit logging
- Restricting connections with IP filtering
- Securing clients and integrations
- Operator privileges
- Troubleshooting
- Some settings are not returned via the nodes settings API
- Authorization exceptions
- Users command fails due to extra arguments
- Users are frequently locked out of Active Directory
- Certificate verification fails for curl on Mac
- SSLHandshakeException causes connections to fail
- Common SSL/TLS exceptions
- Common Kerberos exceptions
- Common SAML issues
- Internal Server Error in Kibana
- Setup-passwords command fails due to connection failure
- Failures due to relocation of the configuration files
- Limitations
- Watcher
- Cross-cluster replication
- Data store architecture
- REST APIs
- API conventions
- Common options
- REST API compatibility
- Autoscaling APIs
- Behavioral Analytics APIs
- Compact and aligned text (CAT) APIs
- cat aliases
- cat allocation
- cat anomaly detectors
- cat component templates
- cat count
- cat data frame analytics
- cat datafeeds
- cat fielddata
- cat health
- cat indices
- cat master
- cat nodeattrs
- cat nodes
- cat pending tasks
- cat plugins
- cat recovery
- cat repositories
- cat segments
- cat shards
- cat snapshots
- cat task management
- cat templates
- cat thread pool
- cat trained model
- cat transforms
- Cluster APIs
- Cluster allocation explain
- Cluster get settings
- Cluster health
- Health
- Cluster reroute
- Cluster state
- Cluster stats
- Cluster update settings
- Nodes feature usage
- Nodes hot threads
- Nodes info
- Prevalidate node removal
- Nodes reload secure settings
- Nodes stats
- Cluster Info
- Pending cluster tasks
- Remote cluster info
- Task management
- Voting configuration exclusions
- Create or update desired nodes
- Get desired nodes
- Delete desired nodes
- Get desired balance
- Reset desired balance
- Cross-cluster replication APIs
- Connector APIs
- Create connector
- Delete connector
- Get connector
- List connectors
- Update connector API key id
- Update connector configuration
- Update connector index name
- Update connector features
- Update connector filtering
- Update connector name and description
- Update connector pipeline
- Update connector scheduling
- Update connector service type
- Create connector sync job
- Cancel connector sync job
- Delete connector sync job
- Get connector sync job
- List connector sync jobs
- Check in a connector
- Update connector error
- Update connector last sync stats
- Update connector status
- Check in connector sync job
- Claim connector sync job
- Set connector sync job error
- Set connector sync job stats
- Data stream APIs
- Document APIs
- Enrich APIs
- EQL APIs
- ES|QL APIs
- Features APIs
- Fleet APIs
- Graph explore API
- Index APIs
- Alias exists
- Aliases
- Analyze
- Analyze index disk usage
- Clear cache
- Clone index
- Close index
- Create index
- Create or update alias
- Create or update component template
- Create or update index template
- Create or update index template (legacy)
- Delete component template
- Delete dangling index
- Delete alias
- Delete index
- Delete index template
- Delete index template (legacy)
- Exists
- Field usage stats
- Flush
- Force merge
- Get alias
- Get component template
- Get field mapping
- Get index
- Get index settings
- Get index template
- Get index template (legacy)
- Get mapping
- Import dangling index
- Index recovery
- Index segments
- Index shard stores
- Index stats
- Index template exists (legacy)
- List dangling indices
- Open index
- Refresh
- Resolve index
- Resolve cluster
- Rollover
- Shrink index
- Simulate index
- Simulate template
- Split index
- Unfreeze index
- Update index settings
- Update mapping
- Index lifecycle management APIs
- Create or update lifecycle policy
- Get policy
- Delete policy
- Move to step
- Remove policy
- Retry policy
- Get index lifecycle management status
- Explain lifecycle
- Start index lifecycle management
- Stop index lifecycle management
- Migrate indices, ILM policies, and legacy, composable and component templates to data tiers routing
- Inference APIs
- Delete inference API
- Get inference API
- Perform inference API
- Create inference API
- Stream inference API
- Update inference API
- AlibabaCloud AI Search inference service
- Amazon Bedrock inference service
- Anthropic inference service
- Azure AI studio inference service
- Azure OpenAI inference service
- Cohere inference service
- Elasticsearch inference service
- ELSER inference service
- Google AI Studio inference service
- Google Vertex AI inference service
- HuggingFace inference service
- Mistral inference service
- OpenAI inference service
- Watsonx inference service
- Info API
- Ingest APIs
- Licensing APIs
- Logstash APIs
- Machine learning APIs
- Machine learning anomaly detection APIs
- Add events to calendar
- Add jobs to calendar
- Close jobs
- Create jobs
- Create calendars
- Create datafeeds
- Create filters
- Delete calendars
- Delete datafeeds
- Delete events from calendar
- Delete filters
- Delete forecasts
- Delete jobs
- Delete jobs from calendar
- Delete model snapshots
- Delete expired data
- Estimate model memory
- Flush jobs
- Forecast jobs
- Get buckets
- Get calendars
- Get categories
- Get datafeeds
- Get datafeed statistics
- Get influencers
- Get jobs
- Get job statistics
- Get model snapshots
- Get model snapshot upgrade statistics
- Get overall buckets
- Get scheduled events
- Get filters
- Get records
- Open jobs
- Post data to jobs
- Preview datafeeds
- Reset jobs
- Revert model snapshots
- Start datafeeds
- Stop datafeeds
- Update datafeeds
- Update filters
- Update jobs
- Update model snapshots
- Upgrade model snapshots
- Machine learning data frame analytics APIs
- Create data frame analytics jobs
- Delete data frame analytics jobs
- Evaluate data frame analytics
- Explain data frame analytics
- Get data frame analytics jobs
- Get data frame analytics jobs stats
- Preview data frame analytics
- Start data frame analytics jobs
- Stop data frame analytics jobs
- Update data frame analytics jobs
- Machine learning trained model APIs
- Clear trained model deployment cache
- Create or update trained model aliases
- Create part of a trained model
- Create trained models
- Create trained model vocabulary
- Delete trained model aliases
- Delete trained models
- Get trained models
- Get trained models stats
- Infer trained model
- Start trained model deployment
- Stop trained model deployment
- Update trained model deployment
- Migration APIs
- Node lifecycle APIs
- Query rules APIs
- Reload search analyzers API
- Repositories metering APIs
- Rollup APIs
- Root API
- Script APIs
- Search APIs
- Search Application APIs
- Searchable snapshots APIs
- Security APIs
- Authenticate
- Change passwords
- Clear cache
- Clear roles cache
- Clear privileges cache
- Clear API key cache
- Clear service account token caches
- Create API keys
- Create or update application privileges
- Create or update role mappings
- Create or update roles
- Bulk create or update roles API
- Bulk delete roles API
- Create or update users
- Create service account tokens
- Delegate PKI authentication
- Delete application privileges
- Delete role mappings
- Delete roles
- Delete service account token
- Delete users
- Disable users
- Enable users
- Enroll Kibana
- Enroll node
- Get API key information
- Get application privileges
- Get builtin privileges
- Get role mappings
- Get roles
- Query Role
- Get service accounts
- Get service account credentials
- Get Security settings
- Get token
- Get user privileges
- Get users
- Grant API keys
- Has privileges
- Invalidate API key
- Invalidate token
- OpenID Connect prepare authentication
- OpenID Connect authenticate
- OpenID Connect logout
- Query API key information
- Query User
- Update API key
- Update Security settings
- Bulk update API keys
- SAML prepare authentication
- SAML authenticate
- SAML logout
- SAML invalidate
- SAML complete logout
- SAML service provider metadata
- SSL certificate
- Activate user profile
- Disable user profile
- Enable user profile
- Get user profiles
- Suggest user profile
- Update user profile data
- Has privileges user profile
- Create Cross-Cluster API key
- Update Cross-Cluster API key
- Snapshot and restore APIs
- Snapshot lifecycle management APIs
- SQL APIs
- Synonyms APIs
- Text structure APIs
- Transform APIs
- Usage API
- Watcher APIs
- Definitions
- Command line tools
- elasticsearch-certgen
- elasticsearch-certutil
- elasticsearch-create-enrollment-token
- elasticsearch-croneval
- elasticsearch-keystore
- elasticsearch-node
- elasticsearch-reconfigure-node
- elasticsearch-reset-password
- elasticsearch-saml-metadata
- elasticsearch-service-tokens
- elasticsearch-setup-passwords
- elasticsearch-shard
- elasticsearch-syskeygen
- elasticsearch-users
- Optimizations
- Troubleshooting
- Fix common cluster issues
- Diagnose unassigned shards
- Add a missing tier to the system
- Allow Elasticsearch to allocate the data in the system
- Allow Elasticsearch to allocate the index
- Indices mix index allocation filters with data tiers node roles to move through data tiers
- Not enough nodes to allocate all shard replicas
- Total number of shards for an index on a single node exceeded
- Total number of shards per node has been reached
- Troubleshooting corruption
- Fix data nodes out of disk
- Fix master nodes out of disk
- Fix other role nodes out of disk
- Start index lifecycle management
- Start Snapshot Lifecycle Management
- Restore from snapshot
- Troubleshooting broken repositories
- Addressing repeated snapshot policy failures
- Troubleshooting an unstable cluster
- Troubleshooting discovery
- Troubleshooting monitoring
- Troubleshooting transforms
- Troubleshooting Watcher
- Troubleshooting searches
- Troubleshooting shards capacity health issues
- Troubleshooting an unbalanced cluster
- Capture diagnostics
- Migration guide
- Release notes
- Elasticsearch version 8.17.1
- Elasticsearch version 8.17.0
- Elasticsearch version 8.16.2
- Elasticsearch version 8.16.1
- Elasticsearch version 8.16.0
- Elasticsearch version 8.15.5
- Elasticsearch version 8.15.4
- Elasticsearch version 8.15.3
- Elasticsearch version 8.15.2
- Elasticsearch version 8.15.1
- Elasticsearch version 8.15.0
- Elasticsearch version 8.14.3
- Elasticsearch version 8.14.2
- Elasticsearch version 8.14.1
- Elasticsearch version 8.14.0
- Elasticsearch version 8.13.4
- Elasticsearch version 8.13.3
- Elasticsearch version 8.13.2
- Elasticsearch version 8.13.1
- Elasticsearch version 8.13.0
- Elasticsearch version 8.12.2
- Elasticsearch version 8.12.1
- Elasticsearch version 8.12.0
- Elasticsearch version 8.11.4
- Elasticsearch version 8.11.3
- Elasticsearch version 8.11.2
- Elasticsearch version 8.11.1
- Elasticsearch version 8.11.0
- Elasticsearch version 8.10.4
- Elasticsearch version 8.10.3
- Elasticsearch version 8.10.2
- Elasticsearch version 8.10.1
- Elasticsearch version 8.10.0
- Elasticsearch version 8.9.2
- Elasticsearch version 8.9.1
- Elasticsearch version 8.9.0
- Elasticsearch version 8.8.2
- Elasticsearch version 8.8.1
- Elasticsearch version 8.8.0
- Elasticsearch version 8.7.1
- Elasticsearch version 8.7.0
- Elasticsearch version 8.6.2
- Elasticsearch version 8.6.1
- Elasticsearch version 8.6.0
- Elasticsearch version 8.5.3
- Elasticsearch version 8.5.2
- Elasticsearch version 8.5.1
- Elasticsearch version 8.5.0
- Elasticsearch version 8.4.3
- Elasticsearch version 8.4.2
- Elasticsearch version 8.4.1
- Elasticsearch version 8.4.0
- Elasticsearch version 8.3.3
- Elasticsearch version 8.3.2
- Elasticsearch version 8.3.1
- Elasticsearch version 8.3.0
- Elasticsearch version 8.2.3
- Elasticsearch version 8.2.2
- Elasticsearch version 8.2.1
- Elasticsearch version 8.2.0
- Elasticsearch version 8.1.3
- Elasticsearch version 8.1.2
- Elasticsearch version 8.1.1
- Elasticsearch version 8.1.0
- Elasticsearch version 8.0.1
- Elasticsearch version 8.0.0
- Elasticsearch version 8.0.0-rc2
- Elasticsearch version 8.0.0-rc1
- Elasticsearch version 8.0.0-beta1
- Elasticsearch version 8.0.0-alpha2
- Elasticsearch version 8.0.0-alpha1
- Dependencies and versions
k-nearest neighbor (kNN) search
editk-nearest neighbor (kNN) search
editA k-nearest neighbor (kNN) search finds the k nearest vectors to a query vector, as measured by a similarity metric.
Common use cases for kNN include:
- Relevance ranking based on natural language processing (NLP) algorithms
- Product recommendations and recommendation engines
- Similarity search for images or videos
Prerequisites
edit-
To run a kNN search, you must be able to convert your data into meaningful vector values. You can create these vectors using a natural language processing (NLP) model in Elasticsearch, or generate them outside Elasticsearch. Vectors can be added to documents as
dense_vector
field values. Queries are represented as vectors with the same dimension.Design your vectors so that the closer a document’s vector is to a query vector, based on a similarity metric, the better its match.
-
To complete the steps in this guide, you must have the following index privileges:
-
create_index
ormanage
to create an index with adense_vector
field -
create
,index
, orwrite
to add data to the index you created -
read
to search the index
-
kNN methods
editElasticsearch supports two methods for kNN search:
-
Approximate kNN using the
knn
search option orknn
query -
Exact, brute-force kNN using a
script_score
query with a vector function
In most cases, you’ll want to use approximate kNN. Approximate kNN offers lower latency at the cost of slower indexing and imperfect accuracy.
Exact, brute-force kNN guarantees accurate results but doesn’t scale well with
large datasets. With this approach, a script_score
query must scan each
matching document to compute the vector function, which can result in slow
search speeds. However, you can improve latency by using a query
to limit the number of matching documents passed to the function. If you
filter your data to a small subset of documents, you can get good search
performance using this approach.
Approximate kNN
editCompared to other types of search, approximate kNN search has specific resource requirements. In particular, all vector data must fit in the node’s page cache for it to be efficient. Please consult the approximate kNN search tuning guide for important notes on configuration and sizing.
To run an approximate kNN search, use the knn
option
to search one or more dense_vector
fields with indexing enabled.
-
Explicitly map one or more
dense_vector
fields. Approximate kNN search requires the following mapping options:-
A
similarity
value. This value determines the similarity metric used to score documents based on similarity between the query and document vector. For a list of available metrics, see thesimilarity
parameter documentation. Thesimilarity
setting defaults tocosine
.
resp = client.indices.create( index="image-index", mappings={ "properties": { "image-vector": { "type": "dense_vector", "dims": 3, "similarity": "l2_norm" }, "title-vector": { "type": "dense_vector", "dims": 5, "similarity": "l2_norm" }, "title": { "type": "text" }, "file-type": { "type": "keyword" } } }, ) print(resp)
response = client.indices.create( index: 'image-index', body: { mappings: { properties: { "image-vector": { type: 'dense_vector', dims: 3, similarity: 'l2_norm' }, "title-vector": { type: 'dense_vector', dims: 5, similarity: 'l2_norm' }, title: { type: 'text' }, "file-type": { type: 'keyword' } } } } ) puts response
const response = await client.indices.create({ index: "image-index", mappings: { properties: { "image-vector": { type: "dense_vector", dims: 3, similarity: "l2_norm", }, "title-vector": { type: "dense_vector", dims: 5, similarity: "l2_norm", }, title: { type: "text", }, "file-type": { type: "keyword", }, }, }, }); console.log(response);
PUT image-index { "mappings": { "properties": { "image-vector": { "type": "dense_vector", "dims": 3, "similarity": "l2_norm" }, "title-vector": { "type": "dense_vector", "dims": 5, "similarity": "l2_norm" }, "title": { "type": "text" }, "file-type": { "type": "keyword" } } } }
-
A
-
Index your data.
POST image-index/_bulk?refresh=true { "index": { "_id": "1" } } { "image-vector": [1, 5, -20], "title-vector": [12, 50, -10, 0, 1], "title": "moose family", "file-type": "jpg" } { "index": { "_id": "2" } } { "image-vector": [42, 8, -15], "title-vector": [25, 1, 4, -12, 2], "title": "alpine lake", "file-type": "png" } { "index": { "_id": "3" } } { "image-vector": [15, 11, 23], "title-vector": [1, 5, 25, 50, 20], "title": "full moon", "file-type": "jpg" } ...
-
Run the search using the
knn
option or theknn
query (expert case).resp = client.search( index="image-index", knn={ "field": "image-vector", "query_vector": [ -5, 9, -12 ], "k": 10, "num_candidates": 100 }, fields=[ "title", "file-type" ], ) print(resp)
response = client.search( index: 'image-index', body: { knn: { field: 'image-vector', query_vector: [ -5, 9, -12 ], k: 10, num_candidates: 100 }, fields: [ 'title', 'file-type' ] } ) puts response
const response = await client.search({ index: "image-index", knn: { field: "image-vector", query_vector: [-5, 9, -12], k: 10, num_candidates: 100, }, fields: ["title", "file-type"], }); console.log(response);
POST image-index/_search { "knn": { "field": "image-vector", "query_vector": [-5, 9, -12], "k": 10, "num_candidates": 100 }, "fields": [ "title", "file-type" ] }
The document _score
is determined by
the similarity between the query and document vector. See
similarity
for more information on how kNN
search scores are computed.
Support for approximate kNN search was added in version 8.0. Before
this, dense_vector
fields did not support enabling index
in the mapping.
If you created an index prior to 8.0 containing dense_vector
fields, then to
support approximate kNN search the data must be reindexed using a new field
mapping that sets index: true
which is the default option.
Tune approximate kNN for speed or accuracy
editTo gather results, the kNN search API finds a num_candidates
number of
approximate nearest neighbor candidates on each shard. The search computes the
similarity of these candidate vectors to the query vector, selecting the k
most similar results from each shard. The search then merges the results from
each shard to return the global top k
nearest neighbors.
You can increase num_candidates
for more accurate results at the cost of
slower search speeds. A search with a high value for num_candidates
considers more candidates from each shard. This takes more time, but the
search has a higher probability of finding the true k
top nearest neighbors.
Similarly, you can decrease num_candidates
for faster searches with
potentially less accurate results.
Approximate kNN using byte vectors
editThe approximate kNN search API supports byte
value vectors in
addition to float
value vectors. Use the knn
option
to search a dense_vector
field with element_type
set to
byte
and indexing enabled.
-
Explicitly map one or more
dense_vector
fields withelement_type
set tobyte
and indexing enabled.resp = client.indices.create( index="byte-image-index", mappings={ "properties": { "byte-image-vector": { "type": "dense_vector", "element_type": "byte", "dims": 2 }, "title": { "type": "text" } } }, ) print(resp)
response = client.indices.create( index: 'byte-image-index', body: { mappings: { properties: { "byte-image-vector": { type: 'dense_vector', element_type: 'byte', dims: 2 }, title: { type: 'text' } } } } ) puts response
const response = await client.indices.create({ index: "byte-image-index", mappings: { properties: { "byte-image-vector": { type: "dense_vector", element_type: "byte", dims: 2, }, title: { type: "text", }, }, }, }); console.log(response);
PUT byte-image-index { "mappings": { "properties": { "byte-image-vector": { "type": "dense_vector", "element_type": "byte", "dims": 2 }, "title": { "type": "text" } } } }
-
Index your data ensuring all vector values are integers within the range [-128, 127].
resp = client.bulk( index="byte-image-index", refresh=True, operations=[ { "index": { "_id": "1" } }, { "byte-image-vector": [ 5, -20 ], "title": "moose family" }, { "index": { "_id": "2" } }, { "byte-image-vector": [ 8, -15 ], "title": "alpine lake" }, { "index": { "_id": "3" } }, { "byte-image-vector": [ 11, 23 ], "title": "full moon" } ], ) print(resp)
response = client.bulk( index: 'byte-image-index', refresh: true, body: [ { index: { _id: '1' } }, { "byte-image-vector": [ 5, -20 ], title: 'moose family' }, { index: { _id: '2' } }, { "byte-image-vector": [ 8, -15 ], title: 'alpine lake' }, { index: { _id: '3' } }, { "byte-image-vector": [ 11, 23 ], title: 'full moon' } ] ) puts response
const response = await client.bulk({ index: "byte-image-index", refresh: "true", operations: [ { index: { _id: "1", }, }, { "byte-image-vector": [5, -20], title: "moose family", }, { index: { _id: "2", }, }, { "byte-image-vector": [8, -15], title: "alpine lake", }, { index: { _id: "3", }, }, { "byte-image-vector": [11, 23], title: "full moon", }, ], }); console.log(response);
POST byte-image-index/_bulk?refresh=true { "index": { "_id": "1" } } { "byte-image-vector": [5, -20], "title": "moose family" } { "index": { "_id": "2" } } { "byte-image-vector": [8, -15], "title": "alpine lake" } { "index": { "_id": "3" } } { "byte-image-vector": [11, 23], "title": "full moon" }
-
Run the search using the
knn
option ensuring thequery_vector
values are integers within the range [-128, 127].resp = client.search( index="byte-image-index", knn={ "field": "byte-image-vector", "query_vector": [ -5, 9 ], "k": 10, "num_candidates": 100 }, fields=[ "title" ], ) print(resp)
response = client.search( index: 'byte-image-index', body: { knn: { field: 'byte-image-vector', query_vector: [ -5, 9 ], k: 10, num_candidates: 100 }, fields: [ 'title' ] } ) puts response
const response = await client.search({ index: "byte-image-index", knn: { field: "byte-image-vector", query_vector: [-5, 9], k: 10, num_candidates: 100, }, fields: ["title"], }); console.log(response);
POST byte-image-index/_search { "knn": { "field": "byte-image-vector", "query_vector": [-5, 9], "k": 10, "num_candidates": 100 }, "fields": [ "title" ] }
Note: In addition to the standard byte array, one can also provide a hex-encoded string value
for the query_vector
param. As an example, the search request above can also be expressed as follows,
which would yield the same results
resp = client.search( index="byte-image-index", knn={ "field": "byte-image-vector", "query_vector": "fb09", "k": 10, "num_candidates": 100 }, fields=[ "title" ], ) print(resp)
response = client.search( index: 'byte-image-index', body: { knn: { field: 'byte-image-vector', query_vector: 'fb09', k: 10, num_candidates: 100 }, fields: [ 'title' ] } ) puts response
const response = await client.search({ index: "byte-image-index", knn: { field: "byte-image-vector", query_vector: "fb09", k: 10, num_candidates: 100, }, fields: ["title"], }); console.log(response);
POST byte-image-index/_search { "knn": { "field": "byte-image-vector", "query_vector": "fb09", "k": 10, "num_candidates": 100 }, "fields": [ "title" ] }
Byte quantized kNN search
editIf you want to provide float
vectors, but want the memory savings of byte
vectors, you can use the
quantization feature. Quantization allows you to provide float
vectors, but
internally they are indexed as byte
vectors. Additionally, the original float
vectors are still retained
in the index.
The default index type for dense_vector
is int8_hnsw
.
To use quantization, you can use the index type int8_hnsw
or int4_hnsw
object in the dense_vector
mapping.
resp = client.indices.create( index="quantized-image-index", mappings={ "properties": { "image-vector": { "type": "dense_vector", "element_type": "float", "dims": 2, "index": True, "index_options": { "type": "int8_hnsw" } }, "title": { "type": "text" } } }, ) print(resp)
response = client.indices.create( index: 'quantized-image-index', body: { mappings: { properties: { "image-vector": { type: 'dense_vector', element_type: 'float', dims: 2, index: true, index_options: { type: 'int8_hnsw' } }, title: { type: 'text' } } } } ) puts response
const response = await client.indices.create({ index: "quantized-image-index", mappings: { properties: { "image-vector": { type: "dense_vector", element_type: "float", dims: 2, index: true, index_options: { type: "int8_hnsw", }, }, title: { type: "text", }, }, }, }); console.log(response);
PUT quantized-image-index { "mappings": { "properties": { "image-vector": { "type": "dense_vector", "element_type": "float", "dims": 2, "index": true, "index_options": { "type": "int8_hnsw" } }, "title": { "type": "text" } } } }
-
Index your
float
vectors.resp = client.bulk( index="quantized-image-index", refresh=True, operations=[ { "index": { "_id": "1" } }, { "image-vector": [ 0.1, -2 ], "title": "moose family" }, { "index": { "_id": "2" } }, { "image-vector": [ 0.75, -1 ], "title": "alpine lake" }, { "index": { "_id": "3" } }, { "image-vector": [ 1.2, 0.1 ], "title": "full moon" } ], ) print(resp)
response = client.bulk( index: 'quantized-image-index', refresh: true, body: [ { index: { _id: '1' } }, { "image-vector": [ 0.1, -2 ], title: 'moose family' }, { index: { _id: '2' } }, { "image-vector": [ 0.75, -1 ], title: 'alpine lake' }, { index: { _id: '3' } }, { "image-vector": [ 1.2, 0.1 ], title: 'full moon' } ] ) puts response
const response = await client.bulk({ index: "quantized-image-index", refresh: "true", operations: [ { index: { _id: "1", }, }, { "image-vector": [0.1, -2], title: "moose family", }, { index: { _id: "2", }, }, { "image-vector": [0.75, -1], title: "alpine lake", }, { index: { _id: "3", }, }, { "image-vector": [1.2, 0.1], title: "full moon", }, ], }); console.log(response);
POST quantized-image-index/_bulk?refresh=true { "index": { "_id": "1" } } { "image-vector": [0.1, -2], "title": "moose family" } { "index": { "_id": "2" } } { "image-vector": [0.75, -1], "title": "alpine lake" } { "index": { "_id": "3" } } { "image-vector": [1.2, 0.1], "title": "full moon" }
-
Run the search using the
knn
option. When searching, thefloat
vector is automatically quantized to abyte
vector.resp = client.search( index="quantized-image-index", knn={ "field": "image-vector", "query_vector": [ 0.1, -2 ], "k": 10, "num_candidates": 100 }, fields=[ "title" ], ) print(resp)
response = client.search( index: 'quantized-image-index', body: { knn: { field: 'image-vector', query_vector: [ 0.1, -2 ], k: 10, num_candidates: 100 }, fields: [ 'title' ] } ) puts response
const response = await client.search({ index: "quantized-image-index", knn: { field: "image-vector", query_vector: [0.1, -2], k: 10, num_candidates: 100, }, fields: ["title"], }); console.log(response);
POST quantized-image-index/_search { "knn": { "field": "image-vector", "query_vector": [0.1, -2], "k": 10, "num_candidates": 100 }, "fields": [ "title" ] }
Since the original float
vectors are still retained in the index, you can optionally use them for re-scoring. Meaning,
you can search over all the vectors quickly using the int8_hnsw
index and then rescore only the top k
results. This
provides the best of both worlds, fast search and accurate scoring.
resp = client.search( index="quantized-image-index", knn={ "field": "image-vector", "query_vector": [ 0.1, -2 ], "k": 15, "num_candidates": 100 }, fields=[ "title" ], rescore={ "window_size": 10, "query": { "rescore_query": { "script_score": { "query": { "match_all": {} }, "script": { "source": "cosineSimilarity(params.query_vector, 'image-vector') + 1.0", "params": { "query_vector": [ 0.1, -2 ] } } } } } }, ) print(resp)
response = client.search( index: 'quantized-image-index', body: { knn: { field: 'image-vector', query_vector: [ 0.1, -2 ], k: 15, num_candidates: 100 }, fields: [ 'title' ], rescore: { window_size: 10, query: { rescore_query: { script_score: { query: { match_all: {} }, script: { source: "cosineSimilarity(params.query_vector, 'image-vector') + 1.0", params: { query_vector: [ 0.1, -2 ] } } } } } } } ) puts response
const response = await client.search({ index: "quantized-image-index", knn: { field: "image-vector", query_vector: [0.1, -2], k: 15, num_candidates: 100, }, fields: ["title"], rescore: { window_size: 10, query: { rescore_query: { script_score: { query: { match_all: {}, }, script: { source: "cosineSimilarity(params.query_vector, 'image-vector') + 1.0", params: { query_vector: [0.1, -2], }, }, }, }, }, }, }); console.log(response);
POST quantized-image-index/_search { "knn": { "field": "image-vector", "query_vector": [0.1, -2], "k": 15, "num_candidates": 100 }, "fields": [ "title" ], "rescore": { "window_size": 10, "query": { "rescore_query": { "script_score": { "query": { "match_all": {} }, "script": { "source": "cosineSimilarity(params.query_vector, 'image-vector') + 1.0", "params": { "query_vector": [0.1, -2] } } } } } } }
Filtered kNN search
editThe kNN search API supports restricting the search using a filter. The search
will return the top k
documents that also match the filter query.
The following request performs an approximate kNN search filtered by the
file-type
field:
resp = client.search( index="image-index", knn={ "field": "image-vector", "query_vector": [ 54, 10, -2 ], "k": 5, "num_candidates": 50, "filter": { "term": { "file-type": "png" } } }, fields=[ "title" ], source=False, ) print(resp)
response = client.search( index: 'image-index', body: { knn: { field: 'image-vector', query_vector: [ 54, 10, -2 ], k: 5, num_candidates: 50, filter: { term: { "file-type": 'png' } } }, fields: [ 'title' ], _source: false } ) puts response
const response = await client.search({ index: "image-index", knn: { field: "image-vector", query_vector: [54, 10, -2], k: 5, num_candidates: 50, filter: { term: { "file-type": "png", }, }, }, fields: ["title"], _source: false, }); console.log(response);
POST image-index/_search { "knn": { "field": "image-vector", "query_vector": [54, 10, -2], "k": 5, "num_candidates": 50, "filter": { "term": { "file-type": "png" } } }, "fields": ["title"], "_source": false }
The filter is applied during the approximate kNN search to ensure
that k
matching documents are returned. This contrasts with a
post-filtering approach, where the filter is applied after the approximate
kNN search completes. Post-filtering has the downside that it sometimes
returns fewer than k results, even when there are enough matching documents.
Approximate kNN search and filtering
editUnlike conventional query filtering, where more restrictive filters typically lead to faster queries,
applying filters in an approximate kNN search with an HNSW index can decrease performance.
This is because searching the HNSW graph requires additional exploration to obtain the num_candidates
that meet the filter criteria.
To avoid significant performance drawbacks, Lucene implements the following strategies per segment:
- If the filtered document count is less than or equal to num_candidates, the search bypasses the HNSW graph and uses a brute force search on the filtered documents.
- While exploring the HNSW graph, if the number of nodes explored exceeds the number of documents that satisfy the filter, the search will stop exploring the graph and switch to a brute force search over the filtered documents.
Combine approximate kNN with other features
editYou can perform hybrid retrieval by providing both the
knn
option and a query
:
resp = client.search( index="image-index", query={ "match": { "title": { "query": "mountain lake", "boost": 0.9 } } }, knn={ "field": "image-vector", "query_vector": [ 54, 10, -2 ], "k": 5, "num_candidates": 50, "boost": 0.1 }, size=10, ) print(resp)
response = client.search( index: 'image-index', body: { query: { match: { title: { query: 'mountain lake', boost: 0.9 } } }, knn: { field: 'image-vector', query_vector: [ 54, 10, -2 ], k: 5, num_candidates: 50, boost: 0.1 }, size: 10 } ) puts response
const response = await client.search({ index: "image-index", query: { match: { title: { query: "mountain lake", boost: 0.9, }, }, }, knn: { field: "image-vector", query_vector: [54, 10, -2], k: 5, num_candidates: 50, boost: 0.1, }, size: 10, }); console.log(response);
POST image-index/_search { "query": { "match": { "title": { "query": "mountain lake", "boost": 0.9 } } }, "knn": { "field": "image-vector", "query_vector": [54, 10, -2], "k": 5, "num_candidates": 50, "boost": 0.1 }, "size": 10 }
This search finds the global top k = 5
vector matches, combines them with the matches from the match
query, and
finally returns the 10 top-scoring results. The knn
and query
matches are combined through a disjunction, as if you
took a boolean or between them. The top k
vector results represent the global nearest neighbors across all index
shards.
The score of each hit is the sum of the knn
and query
scores. You can specify a boost
value to give a weight to
each score in the sum. In the example above, the scores will be calculated as
score = 0.9 * match_score + 0.1 * knn_score
The knn
option can also be used with aggregations
.
In general, Elasticsearch computes aggregations over all documents that match the search.
So for approximate kNN search, aggregations are calculated on the top k
nearest documents. If the search also includes a query
, then aggregations are
calculated on the combined set of knn
and query
matches.
Perform semantic search
editkNN search enables you to perform semantic search by using a previously deployed text embedding model. Instead of literal matching on search terms, semantic search retrieves results based on the intent and the contextual meaning of a search query.
Under the hood, the text embedding NLP model generates a dense vector from the
input query string called model_text
you provide. Then, it is searched
against an index containing dense vectors created with the same text embedding
machine learning model. The search results are semantically similar as learned by the model.
To perform semantic search:
- you need an index that contains the dense vector representation of the input data to search against,
- you must use the same text embedding model for search that you used to create the dense vectors from the input data,
- the text embedding NLP model deployment must be started.
Reference the deployed text embedding model or the model deployment in the
query_vector_builder
object and provide the search query as model_text
:
(...) { "knn": { "field": "dense-vector-field", "k": 10, "num_candidates": 100, "query_vector_builder": { "text_embedding": { "model_id": "my-text-embedding-model", "model_text": "The opposite of blue" } } } } (...)
The natural language processing task to perform. It must be |
|
The ID of the text embedding model to use to generate the dense vectors from
the query string. Use the same model that generated the embeddings from the
input text in the index you search against. You can use the value of the
|
|
The query string from which the model generates the dense vector representation. |
For more information on how to deploy a trained model and use it to create text embeddings, refer to this end-to-end example.
Search multiple kNN fields
editIn addition to hybrid retrieval, you can search more than one kNN vector field at a time:
resp = client.search( index="image-index", query={ "match": { "title": { "query": "mountain lake", "boost": 0.9 } } }, knn=[ { "field": "image-vector", "query_vector": [ 54, 10, -2 ], "k": 5, "num_candidates": 50, "boost": 0.1 }, { "field": "title-vector", "query_vector": [ 1, 20, -52, 23, 10 ], "k": 10, "num_candidates": 10, "boost": 0.5 } ], size=10, ) print(resp)
response = client.search( index: 'image-index', body: { query: { match: { title: { query: 'mountain lake', boost: 0.9 } } }, knn: [ { field: 'image-vector', query_vector: [ 54, 10, -2 ], k: 5, num_candidates: 50, boost: 0.1 }, { field: 'title-vector', query_vector: [ 1, 20, -52, 23, 10 ], k: 10, num_candidates: 10, boost: 0.5 } ], size: 10 } ) puts response
const response = await client.search({ index: "image-index", query: { match: { title: { query: "mountain lake", boost: 0.9, }, }, }, knn: [ { field: "image-vector", query_vector: [54, 10, -2], k: 5, num_candidates: 50, boost: 0.1, }, { field: "title-vector", query_vector: [1, 20, -52, 23, 10], k: 10, num_candidates: 10, boost: 0.5, }, ], size: 10, }); console.log(response);
POST image-index/_search { "query": { "match": { "title": { "query": "mountain lake", "boost": 0.9 } } }, "knn": [ { "field": "image-vector", "query_vector": [54, 10, -2], "k": 5, "num_candidates": 50, "boost": 0.1 }, { "field": "title-vector", "query_vector": [1, 20, -52, 23, 10], "k": 10, "num_candidates": 10, "boost": 0.5 }], "size": 10 }
This search finds the global top k = 5
vector matches for image-vector
and the global k = 10
for the title-vector
.
These top values are then combined with the matches from the match
query and the top-10 documents are returned.
The multiple knn
entries and the query
matches are combined through a disjunction,
as if you took a boolean or between them. The top k
vector results represent the global nearest neighbors across
all index shards.
The scoring for a doc with the above configured boosts would be:
score = 0.9 * match_score + 0.1 * knn_score_image-vector + 0.5 * knn_score_title-vector
Search kNN with expected similarity
editWhile kNN is a powerful tool, it always tries to return k
nearest neighbors. Consequently, when using knn
with
a filter
, you could filter out all relevant documents and only have irrelevant ones left to search. In that situation,
knn
will still do its best to return k
nearest neighbors, even though those neighbors could be far away in the
vector space.
To alleviate this worry, there is a similarity
parameter available in the knn
clause. This value is the required
minimum similarity for a vector to be considered a match. The knn
search flow with this parameter is as follows:
-
Apply any user provided
filter
queries -
Explore the vector space to get
k
vectors -
Do not return any vectors that are further away than the configured
similarity
similarity
is the true similarity before it has been transformed into _score
and boost applied.
For each configured similarity, here is the corresponding inverted _score
function. This is so if you are wanting to filter from a _score
perspective, you can do this minor transformation to correctly reject irrelevant results.
-
l2_norm
:sqrt((1 / _score) - 1)
-
cosine
:(2 * _score) - 1
-
dot_product
:(2 * _score) - 1
-
max_inner_product
:-
_score < 1
:1 - (1 / _score)
-
_score >= 1
:_score - 1
-
Here is an example. In this example we search for the given query_vector
for k
nearest neighbors. However, with
filter
applied and requiring that the found vectors have at least the provided similarity
between them.
resp = client.search( index="image-index", knn={ "field": "image-vector", "query_vector": [ 1, 5, -20 ], "k": 5, "num_candidates": 50, "similarity": 36, "filter": { "term": { "file-type": "png" } } }, fields=[ "title" ], source=False, ) print(resp)
response = client.search( index: 'image-index', body: { knn: { field: 'image-vector', query_vector: [ 1, 5, -20 ], k: 5, num_candidates: 50, similarity: 36, filter: { term: { "file-type": 'png' } } }, fields: [ 'title' ], _source: false } ) puts response
const response = await client.search({ index: "image-index", knn: { field: "image-vector", query_vector: [1, 5, -20], k: 5, num_candidates: 50, similarity: 36, filter: { term: { "file-type": "png", }, }, }, fields: ["title"], _source: false, }); console.log(response);
POST image-index/_search { "knn": { "field": "image-vector", "query_vector": [1, 5, -20], "k": 5, "num_candidates": 50, "similarity": 36, "filter": { "term": { "file-type": "png" } } }, "fields": ["title"], "_source": false }
In our data set, the only document with the file type of png
has a vector of [42, 8, -15]
. The l2_norm
distance
between [42, 8, -15]
and [1, 5, -20]
is 41.412
, which is greater than the configured similarity of 36
. Meaning,
this search will return no hits.
Nested kNN Search
editIt is common for text to exceed a particular model’s token limit and requires chunking before building the embeddings
for individual chunks. When using nested
with dense_vector
, you can achieve nearest
passage retrieval without copying top-level document metadata.
Here is a simple passage vectors index that stores vectors and some top-level metadata for filtering.
resp = client.indices.create( index="passage_vectors", mappings={ "properties": { "full_text": { "type": "text" }, "creation_time": { "type": "date" }, "paragraph": { "type": "nested", "properties": { "vector": { "type": "dense_vector", "dims": 2, "index_options": { "type": "hnsw" } }, "text": { "type": "text", "index": False } } } } }, ) print(resp)
response = client.indices.create( index: 'passage_vectors', body: { mappings: { properties: { full_text: { type: 'text' }, creation_time: { type: 'date' }, paragraph: { type: 'nested', properties: { vector: { type: 'dense_vector', dims: 2, index_options: { type: 'hnsw' } }, text: { type: 'text', index: false } } } } } } ) puts response
const response = await client.indices.create({ index: "passage_vectors", mappings: { properties: { full_text: { type: "text", }, creation_time: { type: "date", }, paragraph: { type: "nested", properties: { vector: { type: "dense_vector", dims: 2, index_options: { type: "hnsw", }, }, text: { type: "text", index: false, }, }, }, }, }, }); console.log(response);
PUT passage_vectors { "mappings": { "properties": { "full_text": { "type": "text" }, "creation_time": { "type": "date" }, "paragraph": { "type": "nested", "properties": { "vector": { "type": "dense_vector", "dims": 2, "index_options": { "type": "hnsw" } }, "text": { "type": "text", "index": false } } } } } }
With the above mapping, we can index multiple passage vectors along with storing the individual passage text.
resp = client.bulk( index="passage_vectors", refresh=True, operations=[ { "index": { "_id": "1" } }, { "full_text": "first paragraph another paragraph", "creation_time": "2019-05-04", "paragraph": [ { "vector": [ 0.45, 45 ], "text": "first paragraph", "paragraph_id": "1" }, { "vector": [ 0.8, 0.6 ], "text": "another paragraph", "paragraph_id": "2" } ] }, { "index": { "_id": "2" } }, { "full_text": "number one paragraph number two paragraph", "creation_time": "2020-05-04", "paragraph": [ { "vector": [ 1.2, 4.5 ], "text": "number one paragraph", "paragraph_id": "1" }, { "vector": [ -1, 42 ], "text": "number two paragraph", "paragraph_id": "2" } ] } ], ) print(resp)
response = client.bulk( index: 'passage_vectors', refresh: true, body: [ { index: { _id: '1' } }, { full_text: 'first paragraph another paragraph', creation_time: '2019-05-04', paragraph: [ { vector: [ 0.45, 45 ], text: 'first paragraph', paragraph_id: '1' }, { vector: [ 0.8, 0.6 ], text: 'another paragraph', paragraph_id: '2' } ] }, { index: { _id: '2' } }, { full_text: 'number one paragraph number two paragraph', creation_time: '2020-05-04', paragraph: [ { vector: [ 1.2, 4.5 ], text: 'number one paragraph', paragraph_id: '1' }, { vector: [ -1, 42 ], text: 'number two paragraph', paragraph_id: '2' } ] } ] ) puts response
const response = await client.bulk({ index: "passage_vectors", refresh: "true", operations: [ { index: { _id: "1", }, }, { full_text: "first paragraph another paragraph", creation_time: "2019-05-04", paragraph: [ { vector: [0.45, 45], text: "first paragraph", paragraph_id: "1", }, { vector: [0.8, 0.6], text: "another paragraph", paragraph_id: "2", }, ], }, { index: { _id: "2", }, }, { full_text: "number one paragraph number two paragraph", creation_time: "2020-05-04", paragraph: [ { vector: [1.2, 4.5], text: "number one paragraph", paragraph_id: "1", }, { vector: [-1, 42], text: "number two paragraph", paragraph_id: "2", }, ], }, ], }); console.log(response);
POST passage_vectors/_bulk?refresh=true { "index": { "_id": "1" } } { "full_text": "first paragraph another paragraph", "creation_time": "2019-05-04", "paragraph": [ { "vector": [ 0.45, 45 ], "text": "first paragraph", "paragraph_id": "1" }, { "vector": [ 0.8, 0.6 ], "text": "another paragraph", "paragraph_id": "2" } ] } { "index": { "_id": "2" } } { "full_text": "number one paragraph number two paragraph", "creation_time": "2020-05-04", "paragraph": [ { "vector": [ 1.2, 4.5 ], "text": "number one paragraph", "paragraph_id": "1" }, { "vector": [ -1, 42 ], "text": "number two paragraph", "paragraph_id": "2" } ] }
The query will seem very similar to a typical kNN search:
resp = client.search( index="passage_vectors", fields=[ "full_text", "creation_time" ], source=False, knn={ "query_vector": [ 0.45, 45 ], "field": "paragraph.vector", "k": 2, "num_candidates": 2 }, ) print(resp)
response = client.search( index: 'passage_vectors', body: { fields: [ 'full_text', 'creation_time' ], _source: false, knn: { query_vector: [ 0.45, 45 ], field: 'paragraph.vector', k: 2, num_candidates: 2 } } ) puts response
const response = await client.search({ index: "passage_vectors", fields: ["full_text", "creation_time"], _source: false, knn: { query_vector: [0.45, 45], field: "paragraph.vector", k: 2, num_candidates: 2, }, }); console.log(response);
POST passage_vectors/_search { "fields": ["full_text", "creation_time"], "_source": false, "knn": { "query_vector": [ 0.45, 45 ], "field": "paragraph.vector", "k": 2, "num_candidates": 2 } }
Note below that even though we have 4 total vectors, we still return two documents. kNN search over nested dense_vectors
will always diversify the top results over the top-level document. Meaning, "k"
top-level documents will be returned,
scored by their nearest passage vector (e.g. "paragraph.vector"
).
{ "took": 4, "timed_out": false, "_shards": { "total": 1, "successful": 1, "skipped": 0, "failed": 0 }, "hits": { "total": { "value": 2, "relation": "eq" }, "max_score": 1.0, "hits": [ { "_index": "passage_vectors", "_id": "1", "_score": 1.0, "fields": { "creation_time": [ "2019-05-04T00:00:00.000Z" ], "full_text": [ "first paragraph another paragraph" ] } }, { "_index": "passage_vectors", "_id": "2", "_score": 0.9997144, "fields": { "creation_time": [ "2020-05-04T00:00:00.000Z" ], "full_text": [ "number one paragraph number two paragraph" ] } } ] } }
What if you wanted to filter by some top-level document metadata? You can do this by adding filter
to your
knn
clause.
filter
will always be over the top-level document metadata. This means you cannot filter based on nested
field metadata.
resp = client.search( index="passage_vectors", fields=[ "creation_time", "full_text" ], source=False, knn={ "query_vector": [ 0.45, 45 ], "field": "paragraph.vector", "k": 2, "num_candidates": 2, "filter": { "bool": { "filter": [ { "range": { "creation_time": { "gte": "2019-05-01", "lte": "2019-05-05" } } } ] } } }, ) print(resp)
response = client.search( index: 'passage_vectors', body: { fields: [ 'creation_time', 'full_text' ], _source: false, knn: { query_vector: [ 0.45, 45 ], field: 'paragraph.vector', k: 2, num_candidates: 2, filter: { bool: { filter: [ { range: { creation_time: { gte: '2019-05-01', lte: '2019-05-05' } } } ] } } } } ) puts response
const response = await client.search({ index: "passage_vectors", fields: ["creation_time", "full_text"], _source: false, knn: { query_vector: [0.45, 45], field: "paragraph.vector", k: 2, num_candidates: 2, filter: { bool: { filter: [ { range: { creation_time: { gte: "2019-05-01", lte: "2019-05-05", }, }, }, ], }, }, }, }); console.log(response);
POST passage_vectors/_search { "fields": [ "creation_time", "full_text" ], "_source": false, "knn": { "query_vector": [ 0.45, 45 ], "field": "paragraph.vector", "k": 2, "num_candidates": 2, "filter": { "bool": { "filter": [ { "range": { "creation_time": { "gte": "2019-05-01", "lte": "2019-05-05" } } } ] } } } }
Now we have filtered based on the top level "creation_time"
and only one document falls within that range.
{ "took": 4, "timed_out": false, "_shards": { "total": 1, "successful": 1, "skipped": 0, "failed": 0 }, "hits": { "total": { "value": 1, "relation": "eq" }, "max_score": 1.0, "hits": [ { "_index": "passage_vectors", "_id": "1", "_score": 1.0, "fields": { "creation_time": [ "2019-05-04T00:00:00.000Z" ], "full_text": [ "first paragraph another paragraph" ] } } ] } }
Nested kNN Search with Inner hits
editAdditionally, if you wanted to extract the nearest passage for a matched document, you can supply inner_hits
to the knn
clause.
When using inner_hits
and multiple knn
clauses, be sure to specify the inner_hits.name
field. Otherwise, a naming clash can occur and fail the search request.
resp = client.search( index="passage_vectors", fields=[ "creation_time", "full_text" ], source=False, knn={ "query_vector": [ 0.45, 45 ], "field": "paragraph.vector", "k": 2, "num_candidates": 2, "inner_hits": { "_source": False, "fields": [ "paragraph.text" ], "size": 1 } }, ) print(resp)
const response = await client.search({ index: "passage_vectors", fields: ["creation_time", "full_text"], _source: false, knn: { query_vector: [0.45, 45], field: "paragraph.vector", k: 2, num_candidates: 2, inner_hits: { _source: false, fields: ["paragraph.text"], size: 1, }, }, }); console.log(response);
POST passage_vectors/_search { "fields": [ "creation_time", "full_text" ], "_source": false, "knn": { "query_vector": [ 0.45, 45 ], "field": "paragraph.vector", "k": 2, "num_candidates": 2, "inner_hits": { "_source": false, "fields": [ "paragraph.text" ], "size": 1 } } }
Now the result will contain the nearest found paragraph when searching.
{ "took": 4, "timed_out": false, "_shards": { "total": 1, "successful": 1, "skipped": 0, "failed": 0 }, "hits": { "total": { "value": 2, "relation": "eq" }, "max_score": 1.0, "hits": [ { "_index": "passage_vectors", "_id": "1", "_score": 1.0, "fields": { "creation_time": [ "2019-05-04T00:00:00.000Z" ], "full_text": [ "first paragraph another paragraph" ] }, "inner_hits": { "paragraph": { "hits": { "total": { "value": 2, "relation": "eq" }, "max_score": 1.0, "hits": [ { "_index": "passage_vectors", "_id": "1", "_nested": { "field": "paragraph", "offset": 0 }, "_score": 1.0, "fields": { "paragraph": [ { "text": [ "first paragraph" ] } ] } } ] } } } }, { "_index": "passage_vectors", "_id": "2", "_score": 0.9997144, "fields": { "creation_time": [ "2020-05-04T00:00:00.000Z" ], "full_text": [ "number one paragraph number two paragraph" ] }, "inner_hits": { "paragraph": { "hits": { "total": { "value": 2, "relation": "eq" }, "max_score": 0.9997144, "hits": [ { "_index": "passage_vectors", "_id": "2", "_nested": { "field": "paragraph", "offset": 1 }, "_score": 0.9997144, "fields": { "paragraph": [ { "text": [ "number two paragraph" ] } ] } } ] } } } } ] } }
Indexing considerations
editFor approximate kNN search, Elasticsearch stores the dense vector values of each segment as an HNSW graph. Indexing vectors for approximate kNN search can take substantial time because of how expensive it is to build these graphs. You may need to increase the client request timeout for index and bulk requests. The approximate kNN tuning guide contains important guidance around indexing performance, and how the index configuration can affect search performance.
In addition to its search-time tuning parameters, the HNSW algorithm has
index-time parameters that trade off between the cost of building the graph,
search speed, and accuracy. When setting up the dense_vector
mapping, you
can use the index_options
argument to adjust
these parameters:
resp = client.indices.create( index="image-index", mappings={ "properties": { "image-vector": { "type": "dense_vector", "dims": 3, "similarity": "l2_norm", "index_options": { "type": "hnsw", "m": 32, "ef_construction": 100 } } } }, ) print(resp)
response = client.indices.create( index: 'image-index', body: { mappings: { properties: { "image-vector": { type: 'dense_vector', dims: 3, similarity: 'l2_norm', index_options: { type: 'hnsw', m: 32, ef_construction: 100 } } } } } ) puts response
const response = await client.indices.create({ index: "image-index", mappings: { properties: { "image-vector": { type: "dense_vector", dims: 3, similarity: "l2_norm", index_options: { type: "hnsw", m: 32, ef_construction: 100, }, }, }, }, }); console.log(response);
PUT image-index { "mappings": { "properties": { "image-vector": { "type": "dense_vector", "dims": 3, "similarity": "l2_norm", "index_options": { "type": "hnsw", "m": 32, "ef_construction": 100 } } } } }
Limitations for approximate kNN search
edit-
When using kNN search in cross-cluster search, the
ccs_minimize_roundtrips
option is not supported. - Elasticsearch uses the HNSW algorithm to support efficient kNN search. Like most kNN algorithms, HNSW is an approximate method that sacrifices result accuracy for improved search speed. This means the results returned are not always the true k closest neighbors.
Approximate kNN search always uses the
dfs_query_then_fetch
search type in order to gather
the global top k
matches across shards. You cannot set the
search_type
explicitly when running kNN search.
Exact kNN
editTo run an exact kNN search, use a script_score
query with a vector function.
-
Explicitly map one or more
dense_vector
fields. If you don’t intend to use the field for approximate kNN, set theindex
mapping option tofalse
. This can significantly improve indexing speed.resp = client.indices.create( index="product-index", mappings={ "properties": { "product-vector": { "type": "dense_vector", "dims": 5, "index": False }, "price": { "type": "long" } } }, ) print(resp)
response = client.indices.create( index: 'product-index', body: { mappings: { properties: { "product-vector": { type: 'dense_vector', dims: 5, index: false }, price: { type: 'long' } } } } ) puts response
const response = await client.indices.create({ index: "product-index", mappings: { properties: { "product-vector": { type: "dense_vector", dims: 5, index: false, }, price: { type: "long", }, }, }, }); console.log(response);
PUT product-index { "mappings": { "properties": { "product-vector": { "type": "dense_vector", "dims": 5, "index": false }, "price": { "type": "long" } } } }
-
Index your data.
POST product-index/_bulk?refresh=true { "index": { "_id": "1" } } { "product-vector": [230.0, 300.33, -34.8988, 15.555, -200.0], "price": 1599 } { "index": { "_id": "2" } } { "product-vector": [-0.5, 100.0, -13.0, 14.8, -156.0], "price": 799 } { "index": { "_id": "3" } } { "product-vector": [0.5, 111.3, -13.0, 14.8, -156.0], "price": 1099 } ...
-
Use the search API to run a
script_score
query containing a vector function.To limit the number of matched documents passed to the vector function, we recommend you specify a filter query in the
script_score.query
parameter. If needed, you can use amatch_all
query in this parameter to match all documents. However, matching all documents can significantly increase search latency.resp = client.search( index="product-index", query={ "script_score": { "query": { "bool": { "filter": { "range": { "price": { "gte": 1000 } } } } }, "script": { "source": "cosineSimilarity(params.queryVector, 'product-vector') + 1.0", "params": { "queryVector": [ -0.5, 90, -10, 14.8, -156 ] } } } }, ) print(resp)
response = client.search( index: 'product-index', body: { query: { script_score: { query: { bool: { filter: { range: { price: { gte: 1000 } } } } }, script: { source: "cosineSimilarity(params.queryVector, 'product-vector') + 1.0", params: { "queryVector": [ -0.5, 90, -10, 14.8, -156 ] } } } } } ) puts response
const response = await client.search({ index: "product-index", query: { script_score: { query: { bool: { filter: { range: { price: { gte: 1000, }, }, }, }, }, script: { source: "cosineSimilarity(params.queryVector, 'product-vector') + 1.0", params: { queryVector: [-0.5, 90, -10, 14.8, -156], }, }, }, }, }); console.log(response);
POST product-index/_search { "query": { "script_score": { "query" : { "bool" : { "filter" : { "range" : { "price" : { "gte": 1000 } } } } }, "script": { "source": "cosineSimilarity(params.queryVector, 'product-vector') + 1.0", "params": { "queryVector": [-0.5, 90.0, -10, 14.8, -156.0] } } } } }
Oversampling and rescoring for quantized vectors
editAll forms of quantization will result in some accuracy loss and as the quantization level increases the accuracy loss will also increase.
Generally, we have found that:
- int8
requires minimal if any rescoring
- int4
requires some rescoring for higher accuracy and larger recall scenarios. Generally, oversampling by 1.5x-2x recovers most of the accuracy loss.
- bbq
requires rescoring except on exceptionally large indices or models specifically designed for quantization. We have found that between 3x-5x oversampling is generally sufficient. But for fewer dimensions or vectors that do not quantize well, higher oversampling may be required.
There are two main ways to oversample and rescore. The first is to utilize the rescore section in the _search
request.
Here is an example using the top level knn
search with oversampling and using rescore
to rerank the results:
resp = client.search( index="my-index", size=10, knn={ "query_vector": [ 0.04283529, 0.85670587, -0.51402352, 0 ], "field": "my_int4_vector", "k": 20, "num_candidates": 50 }, rescore={ "window_size": 20, "query": { "rescore_query": { "script_score": { "query": { "match_all": {} }, "script": { "source": "(dotProduct(params.queryVector, 'my_int4_vector') + 1.0)", "params": { "queryVector": [ 0.04283529, 0.85670587, -0.51402352, 0 ] } } } }, "query_weight": 0, "rescore_query_weight": 1 } }, ) print(resp)
const response = await client.search({ index: "my-index", size: 10, knn: { query_vector: [0.04283529, 0.85670587, -0.51402352, 0], field: "my_int4_vector", k: 20, num_candidates: 50, }, rescore: { window_size: 20, query: { rescore_query: { script_score: { query: { match_all: {}, }, script: { source: "(dotProduct(params.queryVector, 'my_int4_vector') + 1.0)", params: { queryVector: [0.04283529, 0.85670587, -0.51402352, 0], }, }, }, }, query_weight: 0, rescore_query_weight: 1, }, }, }); console.log(response);
POST /my-index/_search { "size": 10, "knn": { "query_vector": [0.04283529, 0.85670587, -0.51402352, 0], "field": "my_int4_vector", "k": 20, "num_candidates": 50 }, "rescore": { "window_size": 20, "query": { "rescore_query": { "script_score": { "query": { "match_all": {} }, "script": { "source": "(dotProduct(params.queryVector, 'my_int4_vector') + 1.0)", "params": { "queryVector": [0.04283529, 0.85670587, -0.51402352, 0] } } } }, "query_weight": 0, "rescore_query_weight": 1 } } }
The number of results to return, note its only 10 and we will oversample by 2x, gathering 20 nearest neighbors. |
|
The number of results to return from the KNN search. This will do an approximate KNN search with 50 candidates
per HNSW graph and use the quantized vectors, returning the 20 most similar vectors
according to the quantized score. Additionally, since this is the top-level |
|
The number of results to rescore, if you want to rescore all results, set this to the same value as |
|
The script to rescore the results. Script score will interact directly with the originally provided float32 vector. |
|
The weight of the original query, here we simply throw away the original score |
|
The weight of the rescore query, here we only use the rescore query |
The second way is to score per shard with the knn query and script_score query . Generally, this means that there will be more rescoring per shard, but this can increase overall recall at the cost of compute.
resp = client.search( index="my-index", size=10, query={ "script_score": { "query": { "knn": { "query_vector": [ 0.04283529, 0.85670587, -0.51402352, 0 ], "field": "my_int4_vector", "num_candidates": 20 } }, "script": { "source": "(dotProduct(params.queryVector, 'my_int4_vector') + 1.0)", "params": { "queryVector": [ 0.04283529, 0.85670587, -0.51402352, 0 ] } } } }, ) print(resp)
const response = await client.search({ index: "my-index", size: 10, query: { script_score: { query: { knn: { query_vector: [0.04283529, 0.85670587, -0.51402352, 0], field: "my_int4_vector", num_candidates: 20, }, }, script: { source: "(dotProduct(params.queryVector, 'my_int4_vector') + 1.0)", params: { queryVector: [0.04283529, 0.85670587, -0.51402352, 0], }, }, }, }, }); console.log(response);
POST /my-index/_search { "size": 10, "query": { "script_score": { "query": { "knn": { "query_vector": [0.04283529, 0.85670587, -0.51402352, 0], "field": "my_int4_vector", "num_candidates": 20 } }, "script": { "source": "(dotProduct(params.queryVector, 'my_int4_vector') + 1.0)", "params": { "queryVector": [0.04283529, 0.85670587, -0.51402352, 0] } } } } }
The number of results to return |
|
The |
|
The number of candidates to use for the initial approximate |
|
The script to score the results. Script score will interact directly with the originally provided float32 vector. |
On this page
- Prerequisites
- kNN methods
- Approximate kNN
- Tune approximate kNN for speed or accuracy
- Approximate kNN using byte vectors
- Byte quantized kNN search
- Filtered kNN search
- Approximate kNN search and filtering
- Combine approximate kNN with other features
- Perform semantic search
- Search multiple kNN fields
- Search kNN with expected similarity
- Nested kNN Search
- Nested kNN Search with Inner hits
- Indexing considerations
- Limitations for approximate kNN search
- Exact kNN
- Oversampling and rescoring for quantized vectors