Elasticsearch version 8.7.0
editElasticsearch version 8.7.0
editAlso see Breaking changes in 8.7.
Known issues
edit-
Shard rebalancing may temporarily unbalance cluster
From 8.6.0 onwards the default shard rebalancing algorithm will compute the final desired balance and then make shard movements to reconcile the current state of the cluster with the desired state. However the order in which the shard movements take place may be skewed towards certain nodes, causing the cluster to become temporarily unbalanced while the reconciliation is ongoing. As always, once a node reaches a disk watermark it will not accept any additional shards, but this skew may result in nodes reaching their disk watermarks more often than expected in normal operation. Once the reconciliation process completes, the cluster will be balanced again.
To avoid this problem, upgrade to 8.8.0 or later.
Breaking changes
edit- Ingest Node
- Indices APIs
-
-
The Resolve index API implementation was adjusted to use the
same index resolution mechanism as other similar APIs, adding support for the
ignore_unavailable
andallow_no_indices
flags and the_all
meta-index. If there are no matching indices then earlier versions of this API would return an empty result with the200 OK
HTTP response code, but from 8.7.0 onwards by default it returns anIndexNotFoundException
with the404 Not Found
HTTP response code. To recover the old behaviour, add the query parameter?ignore_unavailable=true
(#92820).
-
The Resolve index API implementation was adjusted to use the
same index resolution mechanism as other similar APIs, adding support for the
Bug fixes
edit- Aggregations
- Allocation
- Authentication
-
-
Correctly remove domain from realm when rewriting
Authentication
for compatibility with node versions that don’t support domains #93276
-
Correctly remove domain from realm when rewriting
- Authorization
-
- Fix Security’s expression resolver to not remove unavailable but authorized names #92625
- CCR
-
- Deduplicate Heavy CCR Repository CS Requests #91398
- CRUD
- Cluster Coordination
- Data streams
- Geo
- Health
- ILM+SLM
-
- Allow ILM step transition to the phase terminal step #91754
-
Avoiding
BulkProcessor
deadlock in ILMHistoryStore #91238 (issues: #68468, #50440) -
Fixed changing only the
forceMerge
flag inSearchableSnapshotAction
wouldn’t update the policy #93847 - Preventing ILM and SLM runtime state from being stored in a snapshot #92252
- Infra/CLI
- Infra/Core
-
-
Add
jdk.internal.reflect
permission to es codebase #92387 (issue: #92356) - Add checks for exception loops through suppressed exceptions only #93944 (issue: #93943)
- Ensure one-shot wrappers release their delegates #92928
-
Fix
InputStream#readAllBytes
onInputStreamIndexInput
#92680 - Fix indices resolver for datemath with colon #92973
-
Make
FilterStreamInput
less trappy #92422
-
Add
- Infra/Plugins
- Infra/REST API
-
- [Rest Api Compatibility] Format response media type with parameters #92695
- Infra/Scripting
- Infra/Settings
- Ingest Node
-
- Better names and types for ingest stats #93533 (issue: #80763)
- Correctly handle an exception case for ingest failure #92455
- Disable ingest-attachment logging #93878
- Download the geoip databases only when needed #92335 (issue: #90673)
- Forwarding simulate calls to ingest nodes #92171
- Grok returns a list of matches for repeated pattern names #92092 #92586 (issue: #92092)
- Handle a default/request pipeline and a final pipeline with minimal additional overhead #93329 (issues: #92843, #81244, #93118)
- Ingest-attachment module tika dependency versions #93755
- More accurate total ingest stats #91730 (issue: #91358)
- Speed up ingest geoip processors #92372
- Speed up ingest set and append processors #92395
- Machine Learning
-
- Allocate trained models if zone awareness attributes not set #94128 (issue: #94123)
- Fix data counts race condition when starting a datafeed #93324 (issue: #93298)
- Fix tokenization bug when handling normalization in BERT and MPNet #92329
- Free resources correctly when model loading is cancelled #92204
-
Stop the
frequent_items
aggregation reporting a subset when a superset exists #92239 - Use long inference timeout at ingest #93731
- Mapping
- Network
- Recovery
- Rollup
- Search
-
- Add null check for sort fields over collapse fields #94546 (issue: #94407)
- Annotated highlighter does not match when search contains both annotation and annotated term #92920 (issue: #91944)
- Clear field caps index responses on cancelled #93716 (issue: #93029)
- Do not include frozen indices in PIT by default #94377
- Fix NPE thrown by prefix query in strange scenarios #94369
- Fix _id field fetch issue. #94528 (issue: #94515)
-
Fix metadata
_size
when it comes to stored fields extraction #94483 (issue: #94468) -
Fix missing override for matches in
ProfileWeight
#92360 -
Nested path info shouldn’t be added during
copy_to
#93340 (issue: #93117) - Use all profiling events on startup #92087
-
Use keyword analyzer for untokenized fields in
TermVectorsService
#94518 - [Profiling] Adjust handling of last data slice #94283
- [Profiling] Ensure responses are only sent once #93692 (issue: #93691)
- [Profiling] Handle response processing errors #93860
- Snapshot/Restore
- Stats
-
- Avoid capturing cluster state in TBbNA #92255
- TSDB
- Task Management
- Transform
- Vector Search
Enhancements
edit- Aggregations
-
- Optimize composite agg with leading global ordinal value source #92197
- Allocation
-
-
Add
forecasted_write_load
andforecasted_shard_size_in_bytes
to the endpoint #92303 - Expose tier balancing stats via internal endpoint #92199
- Introduce ShardRouting.Role #92668
- Prevalidate node removal API (pt. 2) #91256 (issue: #87776)
- Simulate moves using cluster_concurrent_rebalance=2 #93977
- Unpromotables skip replication and peer recovery #93210
-
Add
- Authentication
- Authorization
- Cluster Coordination
- EQL
-
- EQL Samples: add support for multiple samples per key #91783
- Engine
-
-
Add commits listener for
InternalEngine
andCombinedDeletionPolicy
#92017 - Add primary term supplier to Engine.IndexCommitListener #92101
- Adjust range of allowed percentages of deletes in an index #93188
- Diff the list of filenames that are added by each new commit #92238
- Set a fixed compound file threshold of 1GB #92659
-
Add commits listener for
- Geo
-
- Add methods to H3#hexRing to prevent allocating long arrays #92711
- Add methods to prevent allocating long arrays during child navigation on H3 api #92099
- Add new H3 api method #h3ToNoChildrenIntersecting #91673
- In H3, compute destination point from distance and azimuth using planar 3d math" #93084
- Protect H3 library against integer overflow #92829
- Reduce number of object allocations in H3#h3ToGeoBoundary #91586
-
Speed H3 library by using
FastMath
implementation for trigonometric functions #91839
- Health
- Indices APIs
- Infra/CLI
-
- Scan stable plugins for named components upon install #92528
- Infra/Core
- Infra/Plugins
- Infra/Scripting
- Ingest Node
- Machine Learning
-
- Add identification of multimodal distribution to anomaly explanations #2440
- Add the ability to include and exclude values in Frequent items #92414
-
Better error when
aggregate_metric_double
used in scrolling datafeeds #92232 (issue: #90592) - Implement extension pruning in frequent items to improve runtime #92322
-
Improve
frequent_items
performance using global ordinals #93304 - Improve anomaly detection results indexing speed #92417
- Improve frequent items runtime #93255
- Increase the default timeout for the start trained model deployment API #92328
- Option to delete user-added annotations for the reset/delete job APIs #91698 (issue: #74310)
- Persist data counts and datafeed timing stats asynchronously #93000
- Remove the PyTorch inference work queue as now handled in Elasticsearch #2456
- Text Embedding search #93531
- Upgrade PyTorch to version 1.13.1 #2430
- Mapping
-
-
Switch to Lucene’s new
IntField/LongField/FloatField/DoubleField
#93165
-
Switch to Lucene’s new
- Monitoring
- Network
-
- Deserialize responses on the handling thread-pool #91367
- Performance
-
- Add vector distance scoring to micro benchmarks #92340
- Query Languages
-
- Introduce parameterized rule and executor #92428
- Recovery
-
- Make clean up files step configurable for peer-recovery of replicas #92490
- Search
-
- Access term dictionary more efficiently #92269
-
Add
term
query support torank_features
mapped field #93247 -
Add new
query_vector_builder
option to knn search clause #93331 - Add profiling plugin #91640
- Enable profiling plugin by default #92787
- Get stackframes and executables more concurrently #93559
- Improve the false positive rate of the bloom filter by setting 7 hash functions #93283
- Increase the number of threads of GET threadpool #92309
- Instrument Weight#count in ProfileWeight #85656 (issue: #85203)
- Reduce memory usage of match all bitset #92777
- Runtime fields to optionally ignore script errors #92380
- Speed up retrieval of data for flamegraphs #93448
- Support retrieving inlined stack frames #92863
- [Profiling] Reduce GC pressure #93590
- Security
- Snapshot/Restore
-
-
Make
RecoveryPlannerService
optional #92489
-
Make
- TSDB
-
-
Enable bloom filter for
_id
field in tsdb indices #92115 - Improve downsampling performance by removing map lookups #92494 (issue: #90226)
- Minor TSDB parsing speedup #92276
- Skip duplicate checks on segments that don’t contain the document’s timestamp #92456
-
Support
fields
in synthetic source in last cases #91595
-
Enable bloom filter for
- Task Management
- Transform
- Vector Search
- Watcher
-
- Add ability for Watcher’s webhook actions to send additional header #93426