IMPORTANT: No additional bug fixes or documentation updates
will be released for this version. For the latest information, see the
current release documentation.
Elasticsearch version 7.4.0
editElasticsearch version 7.4.0
editAlso see Breaking changes in 7.4.
Known issues
edit- Applying deletes or updates on an index after it has been shrunk may corrupt the index. In order to prevent this issue, it is recommended to stop shrinking read-write indices. For read-only indices, it is recommended to force-merge indices after shrinking, which significantly reduces the likeliness of this corruption in the case that deletes/updates would be applied by mistake. This bug is fixed in Elasticsearch 7.7 and later versions. More details can be found on the corresponding issue.
- Activating the search slow log should be avoided in this version. Any attempt to log a slow search can throw an AIOOBE due to a bug that performs concurrent modifications on a shared byte array. (issue: #48358)
-
Indices created in 6.x with
date
anddate_nanos
fields using formats that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results. https://github.com/elastic/elasticsearch/pull/52555 This is fixed in Elasticsearch 7.7 and later versions.
Breaking changes
editBreaking Java changes
edit- Geo
- Network
-
- Stop Recreating Wrapped Handlers in RestController #44964
Deprecations
edit- Geo
- Infra/Core
- Infra/Settings
- Machine Learning
New features
edit- Aggregations
- Analysis
- Authentication
- Authorization
-
- Add granular privileges for API keys #42020
- Features/ILM
- Features/Watcher
- Geo
- Machine Learning
-
- Allow the user to specify query in Evaluate Data Frame request #45775 (issue: #45729)
- Call the new _estimate_memory_usage API endpoint on data frame analytics _start #45536 (issues: #44699, #45544)
- HLRC for memory usage estimation API #45531 (issues: #44699, #45188)
- Implement ml/data_frame/analytics/_estimate_memory_usage API endpoint #45188 (issue: #44699)
Enhancements
edit- Aggregations
- Allocation
- Analysis
- Authentication
- Authorization
- CCR
-
- Include leases in error message when operations no longer available #45681
- CRUD
- Cluster Coordination
- Distributed
- Docs Infrastructure
-
- add clarification around TESTSETUP docu and error message #43306
- Engine
-
- Flush engine after big merge #46066
- Do sync before closeIntoReader when rolling generation to improve index performance #45765 (issue: #45371)
- Refactor index engines to manage readers instead of searchers #43860
- Async IO Processor release before notify #43682
- Enable indexing optimization using sequence numbers on replicas #43616 (issue: #34099)
- Features/ILM
- Features/Indices APIs
- Features/Ingest
-
- Fix IngestService to respect original document content type #45799
- Ingest Attachment: Upgrade tika to v1.22 #45575
- Retrieve processors instead of checking existence #45354
- Add ingest processor existence helper method #45156
- Change the ingest simulate api to not include dropped documents #44161 (issue: #36150)
- Features/Java High Level REST Client
-
- Add XContentType as parameter to HLRC ART#createServerTestInstance #46036 (issue: #45970)
- Add CloseIndexResponse to HLRC #44349 (issue: #39687)
- Add mapper-extras and the RankFeatureQuery in the hlrc #43713 (issue: #43634)
- removing background state update of Request object by RequestConverte… #40156 (issue: #39666)
- Add delete aliases API to the high-level REST client #32909 (issue: #27205)
- Features/Watcher
-
- Add SSL/TLS settings for watcher email #45272 (issue: #30307)
- Watcher reporting: add email warning if CSV attachment contains values that may be interperted as formulas #44460
- Watcher add stopped listener #43939 (issue: #42409)
- Improve CryptoService error message on missing secure file #43623 (issue: #43619)
- Geo
- Infra/Circuit Breakers
-
- Fix G1 GC default IHOP #46169
- Infra/Core
- Infra/Packaging
- Infra/Plugins
- Infra/Scripting
- Infra/Settings
- Machine Learning
-
- Support boolean fields for data frame analytics #46037
- Add description to data frame analytics #45774
- Add regression analysis to data frame analytics #45292
- Introduce formal node ML role #45174 (issues: #29943, #43175)
- Improve CSV header row detection in find_file_structure #45099 (issue: #45047)
- Outlier detection should only fetch docs that have the analyzed fields #44944
- Persist DatafeedTimingStats with RefreshPolicy.NONE by default #44940 (issue: #44792)
- Add result_type field to TimingStats and DatafeedTimingStats documents #44812
- Implement exponential average search time per hour statistics. #44683 (issue: #29857)
- Add r_squared eval metric to regression #44248
- Adds support for regression.mean_squared_error to eval API #44140
- Add DatafeedTimingStats.average_search_time_per_bucket_ms and TimingStats.total_bucket_processing_time_ms stats #44125 (issue: #29857)
- Add DatafeedTimingStats to datafeed GetDatafeedStatsAction.Response #43045 (issue: #29857)
- Network
-
- Better logging for TLS message on non-secure transport channel #45835 (issue: #32688)
- Asynchronously connect to remote clusters #44825 (issue: #40150)
- Improve errors when TLS files cannot be read #44787 (issue: #43079)
- Add per-socket keepalive options #44055
- Move ConnectionManager to async APIs #42636
- Ranking
- Recovery
-
- Ignore translog retention policy if soft-deletes enabled #45473 (issue: #45136)
- Only retain reasonable history for peer recoveries #45208 (issue: #41536)
- Use index for peer recovery instead of translog #45136 (issues: #38904, #41536, #42211)
- Trim local translog in peer recovery #44756
- Make peer recovery send file chunks async #44468 (issues: #36195, #44040)
- SQL
-
- SQL: Support queries with HAVING over SELECT #46709 (issue: #37051)
- SQL: Break TextFormatter/Cursor dependency #45613 (issue: #45516)
- SQL: remove deprecated use of "interval" from date_histogram usage #45501 (issue: #43922)
- SQL: use hasValue() methods from Elasticsearch’s InspectionHelper classes #44745 (issue: #35745)
- Switch from using docvalue_fields to extracting values from _source #44062 (issue: #41852)
- Search
- Security
- Snapshot/Restore
- Task Management
- Transforms
-
- Add update transform api endpoint #45154 (issue: #43438)
- Add support for bucket_selector #44718 (issues: #43744, #44557)
- Add force delete #44590 (issue: #43961)
- Add dynamic cluster setting for failure retries #44577
- Add optional defer_validation param to PUT #44455 (issue: #43439)
- Add support for geo_bounds aggregation #44441
- Add a frequency option to transform config, default 1m #44120
Bug fixes
edit- Aggregations
- Allocation
- Analysis
- Authentication
-
- Fallback to realm authc if ApiKey fails #46538
- Enforce realm name uniqueness #46253
- Allow empty token endpoint for implicit flow #45038
- Do not use scroll when finding duplicate API key #45026
- Fix broken short-circuit in getUnlicensedRealms #44399
- Fix X509AuthenticationToken principal #43932 (issues: #34396, #43796)
- Authorization
- CCR
- CRUD
- Cluster Coordination
- Distributed
-
- Fix false positive out of sync warning in synced-flush #46576 (issues: #28464, #30244)
- Suppress warning logs from background sync on relocated primary #46247 (issues: #40800, #42241)
- Ensure AsyncTask#isScheduled remain false after close #45687 (issue: #45576)
- Update translog checkpoint after marking operations as persisted #45634 (issue: #29161)
- Fix clock used in update requests #45262 (issue: #45254)
- Restore DefaultShardOperationFailedException’s reason during deserialization #45203
- Use IndicesModule named writables in elasticsearch-shard tool #45036 (issue: #44628)
- Engine
-
- Handle delete document level failures #46100 (issue: #46083)
- Handle no-op document level failures #46083
- Remove leniency during replay translog in peer recovery #44989
- Throw TranslogCorruptedException in more cases #44217
- Fail engine if hit document failure on replicas #43523 (issues: #40435, #43228)
- Features/ILM
- Features/Indices APIs
- Features/Ingest
- Features/Java High Level REST Client
-
- [HLRC] Send min_score as query string parameter to the count API #46829 (issue: #46474)
- HLRC multisearchTemplate forgot params #46492 (issue: #46488)
- terminateAfter added to the RequestConverter #46474 (issue: #46446)
- [Closes #44045] Added slices parameter when submitting reindex request via Java high level REST client #45690 (issue: #44045)
- HLRC: Fix + Not Correctly Encoded in GET Req. #33164 (issue: #33077)
- Features/Watcher
- Geo
- Highlighting
- Infra/Core
- Infra/Logging
- Infra/Packaging
- Infra/Plugins
- Infra/REST API
- Infra/Scripting
- Infra/Settings
- Machine Learning
-
- Fix two datafeed flush lockup bugs #46982
- Throw an error when a datafeed needs CCS but it is not enabled for the node #46044 (issue: #46025)
- Handle "null" value of Estimate memory usage API response gracefully. #45726 (issue: #44699)
- Remove timeout on waiting for data frame analytics result processor to complete #45724 (issue: #45723)
- Check dest index is empty when starting data frame analytics #45094
- Catch any error thrown while closing data frame analytics process #44958
- Treat PostDataActionResponse.DataCounts.bucketCount as incremental rather than absolute (total). #44803 (issue: #44792)
- Treat big changes in searchCount as significant and persist the document after such changes #44413 (issues: #44196, #44335)
- Update .ml-config mappings before indexing job, datafeed or data frame analytics config #44216 (issue: #44263)
- Wait for .ml-config primary before assigning persistent tasks #44170 (issue: #44156)
- Fix ML memory tracker lockup when inner step fails #44158 (issue: #44156)
- Fix datafeed checks when a concrete remote index is present #43923 (issue: #42113)
-
Rename outlier detection method values
knn
andtnn
todistance_kth_nn
anddistance_knn
respectively to match the API. #598 -
Fix occasional (non-deterministic) reinitialisation of modeling for the
lat_long
function. #641
- Mapping
-
- Make sure to validate the type before attempting to merge a new mapping. #45157 (issues: #29316, #43012)
- Ensure field caps doesn’t error on rank feature fields. #44370 (issue: #44330)
- Prevent types deprecation warning for indices.exists requests #43963 (issue: #43905)
- Add include_type_name in indices.exitst REST API spec #43910 (issue: #43905)
- Network
- Recovery
- Rollup
-
- Fix Rollup job creation to work with templates #43943
- SQL
-
- SQL: Properly handle indices with no/empty mapping #46775 (issue: #46757)
- SQL: improve ResultSet behavior when no rows are available #46753 (issue: #46750)
- SQL: use the correct data type for types conversion #46574 (issue: #46090)
- SQL: Fix issue with common type resolution #46565 (issue: #46551)
- SQL: fix scripting for grouped by datetime functions #46421 (issue: #40241)
- SQL: Use null schema response #46386 (issue: #46381)
- SQL: Fix issue with IIF function when condition folds #46290 (issue: #46268)
- SQL: Fix issue with DataType for CASE with NULL #46173 (issue: #46032)
- SQL: adds format parameter to range queries for constant date comparisons #45326 (issue: #45139)
- SQL: uniquely named inner_hits sections for each nested field condition #45039 (issues: #33080, #44544)
- SQL: fix URI path being lost in case of hosted ES scenario #44776 (issue: #44721)
- SQL: change the size of the list of concrete indices when resolving multiple indices #43878 (issue: #43876)
- SQL: handle double quotes escaping #43829 (issue: #43810)
- SQL: add pretty printing to JSON format #43756
- SQL: handle SQL not being available in a more graceful way #43665 (issue: #41279)
- Search
-
- Multi-get requests should wait for search active #46283 (issue: #27500)
- Ensure top docs optimization is fully disabled for queries with unbounded max scores. #46105 (issue: #45933)
- Disallow partial results when shard unavailable #45739 (issue: #42612)
- Prevent Leaking Search Tasks on Exceptions in FetchSearchPhase and DfsQueryPhase #45500
- Fix an NPE when requesting inner hits and _source is disabled. #44836 (issue: #43517)
- Don’t use index_phrases on graph queries #44340 (issue: #43976)
- Security
-
- Initialize document subset bit set cache used for DLS #46211 (issue: #45147)
- Fix .security-* indices auto-create #44918
- SecurityIndexManager handle RuntimeException while reading mapping #44409
- Do not swallow I/O exception getting authentication #44398 (issues: #44172, #44397)
- Use system context for looking up connected nodes #43991 (issue: #43974)
- Snapshot/Restore
-
- Fix Bug in Snapshot Status Response Timestamps #46919 (issue: #46913)
- GCS deleteBlobsIgnoringIfNotExists should catch StorageException #46832 (issue: #46772)
- Fix TransportSnapshotsStatusAction ThreadPool Use #45824
- Stop Executing SLM Policy Transport Action on Snapshot Pool #45727 (issue: #45594)
- Check again on-going snapshots/restores of indices before closing #43873
- Make Timestamps Returned by Snapshot APIs Consistent #43148 (issue: #43074)
- Recursively Delete Unreferenced Index Directories #42189 (issue: #13159)
- Task Management
-
- Catch AllocatedTask registration failures #45300
- Transforms
-
- Use field_caps API for mapping deduction #46703 (issue: #46694)
- Fix off-by-one error in checkpoint operations_behind #46235
- Moves failure state transition for MT safety #45676 (issue: #45664)
- Fix _start?force=true bug #45660
- Fix failure state transitions and race condition #45627 (issues: #45562, #45609)
- Fix starting a batch data frame after stopping at runtime #45340 (issues: #44219, #45339)
- Fix null aggregation handling in indexer #45061 (issue: #44906)
- Unify validation exceptions between PUT and _preview #44983 (issue: #44953)
- Treat bulk index failures as an indexing failure #44351 (issue: #44101)
- Prevent task from attempting to run when failed #44239 (issue: #44121)
- Respond with 409 status code when failing _stop #44231 (issue: #44103)
- Add index validations to _start data frame transform #44191 (issue: #44104)
- Data frame task failure does not make a 500 response #44058 (issue: #44011)
- Audit message missing for autostop #43984 (issue: #43977)