« Elasticsearch version 8.4.1 Elasticsearch version 8.3.3 »
Elastic Docs ›Elasticsearch Guide [8.15] ›Release notes

Elasticsearch version 8.4.0

edit

Elasticsearch version 8.4.0

edit

Also see Breaking changes in 8.4.

Known issues

edit

  • Machine learning datafeeds cannot be listed if any are not modified since version 6.x

    If you have a datafeed that was created in version 5.x or 6.x and has not been updated since 7.0, it is not possible to list datafeeds in 8.4 and 8.5. This means that anomaly detection jobs cannot be managed using Kibana. This issue is fixed in 8.6.0.

    If you upgrade to 8.4 or 8.5 with such a datafeed, you need to work around the problem by updating each datafeed’s authorization information using these steps.

  • Orchestrators which use a settings.json file to configure Elasticsearch may encounter deadlocks during master elections (issue: #92812)

    To resolve the deadlock, remove the settings.json file and restart the affected node.

  • Parsing a request when the last element in an array is filtered out (for instance using _source_includes) fails. This is due to a bug in Jackson parser. Fixed in Elasticsearch 8.6.1 (#91456)

  • When the ingest attachment processor is used, the interaction of Tika with log4j 2.18.0 and higher (introduced in Elasticsearch 8.4.0) results in excessive logging. This logging is so excessive that it can lead to cluster instability, to the point where the cluster is unusable and nodes must be restarted. (issue: #91964). This issue is fixed in Elasticsearch 8.7.0 (#93878)

    To resolve the issue, upgrade to 8.7.0 or higher.

Bug fixes

edit
Aggregations
  • Fix multi-value handling in composite agg #88638
  • Fix: extract matrix stats using bucket_selector buckets_path #88271 (issue: #87454)
  • Make the metric in the buckets_path parameter optional #87220 (issue: #72983)
  • Propagate alias filters to significance aggs filters #88221 (issue: #81585)
Allocation
  • Clamp auto-expand replicas to the closest value #87505 (issue: #84788)
  • Prevent re-balancing using outdated node weights in some cases #88385 (issue: #88384)
  • Remove any existing read_only_allow_delete index blocks when cluster.routing.allocation.disk.threshold_enabled is set to false #87841 (issue: #86383)
  • Replace health request with a state observer #88641
Authentication
  • Fix unique realm name check to cover default realms #87999
Authorization
  • Add rollover permissions for remote_monitoring_agent #87717 (issue: #84161)
Autoscaling
  • Autoscaling during shrink #88292 (issue: #85480)
  • Do not include desired nodes in snapshots #87695
Cluster Coordination
  • Improve rejection of ambiguous voting config name #89239
Data streams
  • Fix renaming data streams with CCR replication #88875 (issue: #81751)
Distributed
  • Fixed NullPointerException on bulk request #88385
EQL
  • Avoid attempting PIT close on PIT open failure #87498
  • Improve EQL Sequence circuit breaker precision #88538 (issue: #88300)
Geo
  • Geo_line aggregation returns a geojson point when the resulting line has only one point #89199 (issue: #85748)
  • Sort ranges in geo_distance aggregation #89154 (issue: #89147)
Health
  • Fix NPE when checking if the last snapshot was success #88811
  • Fixing a version check for master stability functionality #89322
  • Fixing internal action names #89182
  • Using the correct connection to fetch remote master history #87299
Highlighting
  • Handle ordering in plain highlighter for multiple inputs #87414 (issue: #87210)
ILM+SLM
  • Batch ILM move to retry step task update #86759
Infra/CLI
  • Quote paths with whitespace in Windows service CLIs #89072 (issue: #89043)
Infra/Core
  • Always close directory streams #88560
  • Delete invalid settings for system indices #88903 (issue: #88324)
  • Disallow three-digit minor and revision versions #87338
  • Handle snapshot restore in file settings #89321 (issue: #89183)
  • System indices ignore all user templates #87260 (issues: #42508, #74271)
Infra/Node Lifecycle
  • Fix message for stalled shutdown #89254
Infra/Plugins
  • Disable URL connection caching in SPIClassIterator #88586 (issue: #88275)
Infra/Scripting
  • Script: UpdateByQuery can read doc version if requested #88740
Machine Learning
  • Address potential bug where trained models get stuck in starting after being allocated to node #88945
  • Fix BERT and MPNet tokenization bug when handling unicode accents #88907 (issue: #88900)
  • Fix NLP question_answering task when best answer is only one token #88347
  • Include start params in _stats for non-started model deployments #89091
  • Fix minor tokenization bug when using fill_mask task with roberta tokenizer #88825
  • Fix potential cause of classification and regression job failures #2385
Mapping
  • Assign the right path to objects merged when parsing mappings #89389 (issue: #88573)
  • Don’t modify source map when parsing composite runtime field #89114
Network
  • Ensure that the extended socket options TCP_KEEPXXX are available #88935 (issue: #88897)
SQL
Search
  • Fix: use status code 500 for aggregation reduce phase errors if no shard failed #88551 (issue: #20004)
  • Override bulk visit methods of exitable point visitor #82120
Security
  • Ensure secureString remain open when reloading secure settings #88922
Snapshot/Restore
  • Fix queued snapshot assignments after partial snapshot fails due to delete #88470 (issue: #86724)
Transform
  • Handle update error correctly #88619

Deprecations

edit
Vector Search
  • Deprecate the _knn_search endpoint #88828

Enhancements

edit
Aggregations
  • Adding cardinality support for random_sampler agg #86838
  • Minor RangeAgg optimization #86935 (issue: #84262)
  • Speed counting filters/range/date_histogram aggs #81322
  • Update bucket metric pipeline agg paths to allow intermediate single bucket and bucket qualified multi-bucket aggs #85729
Allocation
  • Add debug information to ReactiveReason about assigned and unassigned shards #86132 (issue: #85243)
  • Optimize log cluster health performance. #87723
  • Use desired nodes during data tier allocation decisions #87735
Audit
  • Audit API key ID when create or grant API keys #88456
  • Include API key metadata in audit log when an API key is created, granted, or updated #88642
  • Updatable API keys - logging audit trail event #88276
  • User Profile - audit support for security domain #87097
Authentication
  • If signature validation fails, reload JWKs and retry if new JWKs are found #88023
Authorization
  • App permissions with action patterns do not retrieve privileges #85455
  • Cancellable Profile Has Privilege check #87224
  • Return action denied error when user with insufficient privileges (manage_own_api_key) attempts a grant API key request #87461 (issue: #87438)
  • Update indices permissions to Enterprise Search service account #88703
Autoscaling
  • Add processors to autoscaling capacity response #87895
  • Keep track of desired nodes status in cluster state #87474
Cluster Coordination
  • Deduplicate mappings in persisted cluster state #88479
  • Expose segment details in PCSS debug log #87412
  • Periodic warning for 1-node cluster w/ seed hosts #88013 (issue: #85222)
  • Report overall mapping size in cluster stats #87556
Data streams
  • Give doc-value-only mappings to numeric fields on metrics templates #87100
Distributed
  • Adding the ability to register a PeerFinderListener to Coordinator #88626
  • Make Desired Nodes API operator-only #87778 (issue: #87777)
  • Support "dry run" mode for updating Desired Nodes #88305
FIPS
  • Log warning when hash function used by cache is not recommended in FIPS mode #86740
  • Log warning when hashers for stored API keys or service tokens are not compliant with FIPS #87363
Geo
  • Optimize geogrid aggregations for singleton points #87439
  • Support cartesian shape with doc values #88487
  • Use a faster but less accurate log algorithm for computing Geotile Y coordinate #87515
  • Use faster maths to project WGS84 to mercator #88231
Health
  • Add health user action for unhealthy SLM policy failure counts #88523
  • Adding a transport action to get cluster formation info #87306
  • Adding additional capability to the master_is_stable health indicator service #87482
  • Creating a transport action for the CoordinationDiagnosticsService #87984
  • Move the master stability logic into its own service separate from the HealthIndicatorService #87672
  • Polling cluster formation state for master-is-stable health indicator #88397
  • Remove cluster block preflight check from health api #87520 (issue: #87464)
ILM+SLM
  • Add min_* conditions to rollover #83345
  • Track the count of failed invocations since last successful policy snapshot #88398
Infra/Core
  • Improve console exception messages #87942
  • Print full exception when console is non-interactive #88297
  • Stop making index read-only when executing force merge index lifecycle management action #81162 (issue: #81162)
  • Stream input and output support for optional collections #88127
  • Update version of internal http client #87491
Infra/Logging
  • Catch an exception when formatting a string fails #87132
Infra/Scripting
  • Script: Add Metadata to ingest context #87309
  • Script: Metadata for update context #88333
Infra/Settings
  • Convert disk watermarks to RelativeByteSizeValues #88719
Ingest
  • Allow pipeline processor to ignore missing pipelines #87354
  • Move the ingest attachment processor to the default distribution #87989
  • Only perform ensureNoSelfReferences check during ingest when needed #87352 (issue: #87335)
  • Removing BouncyCastle dependencies from ingest-attachment plugin #88031
Machine Learning
  • Add authorization info to ML config listings #87884
  • Add deployed native models to inference_stats in trained model stats response #88187
  • Add inference cache hit count to inference node stats #88807
  • Add new cache_size parameter to trained_model deployments API #88450
  • Expand allowed NER labels to be any I-O-B tagged labels #87091
  • Improve scalability of NLP models #87366
  • Indicate overall deployment failure if all node routes are failed #88378
  • New frequent_items aggregation #83055
  • Fairer application of size penalty for model selection for training classification and regression models #2291
  • Accelerate training for data frame analytics by skipping fine parameter tuning if it is unnecessary #2298
  • Address some causes of high runtimes training regression and classification models on large data sets with many features #2332
  • Add caching for PyTorch inference #2305
  • Improve accuracy of anomaly detection median estimation #2367 (issue: #2364)
Mapping
  • Enable synthetic source support on constant keyword fields #88603
  • Speed up NumberFieldMapper #85688
Monitoring
  • JvmService use SingleObjectCache #87236
Network
  • Allow start cluster with unreachable remote clusters #87298
  • Increase http.max_header_size default to 16kb #88725 (issue: #88501)
Query Languages
  • Add support for VERSION field type in SQL and EQL #87590 (issue: #83375)
Rollup
  • [TSDB] Add Kahan support to downsampling summation #87554
SQL
  • Implement support for partial search results in SQL CLI #86982 (issue: #86082)
  • Update Tableau connector to use connection dialog v2 #88462
Search
  • Add mapping stats for indexed dense_vectors #86859
  • Improve error when sorting on incompatible types #88399 (issue: #73146)
  • Support kNN vectors in disk usage action #88785 (issue: #84801)
Security
  • Add setting for tcp_keepalive for oidc back-channel #87868
  • Support run_as another user when granting API keys #88335
  • Support exists query for API key query #87229
  • Updatable API keys - REST API spec and tests #88270
  • Updatable API keys - noop check #88346
Snapshot/Restore
  • INFO logging of snapshot restore and completion #88257 (issue: #86610)
  • Make snapshot deletes not block the repository during data blob deletes #86514
  • Retry after all S3 get failures that made progress #88015 (issue: #87243)
  • Speed up creating new IndexMetaDataGenerations without removed snapshots #88344
  • Update HDFS Repository to HDFS 3.3.3 #88039
Stats
  • Sort ingest pipeline stats by use #88035
TLS
  • Add issuer to GET _ssl/certificates #88445
Transform
  • Add authorization info to transform config listings #87570
  • Implement per-transform num_failure_retries setting #87361

New features

edit
Authentication
  • Support updates of API key attributes (single operation route) #88186
Health
  • Master stability health indicator part 1 (when a master has been seen recently) #86524
  • Remove help_url,rename summary to symptom, and user_actions to diagnosis #88553 (issue: #88474)
Infra/Core
  • File Settings Service #88329
Infra/Logging
  • Stable logging API - the basic use case #86612
Machine Learning
  • Make composite aggs in datafeeds Generally Available #88589
Search
  • Add mode option to _source field mapper #88211
TSDB
  • TSDB: Implement downsampling ILM Action for time-series indices #87269 (issue: #68609)
Vector Search
  • Integrate ANN into _search endpoint #88694 (issue: #87625)

Upgrades

edit
Infra/Core
  • Upgrade to Log4J 2.18.0 #88237
Network
  • Upgrade to Netty 4.1.77 #86630
« Elasticsearch version 8.4.1 Elasticsearch version 8.3.3 »