New

The executive guide to generative AI

Read more
About usPartnersSupport|Login
A newer version is available. For the latest information, see the current release documentation.
« Elasticsearch version 7.5.1 Elasticsearch version 7.4.2 »
Elastic Docs ›Elasticsearch Guide [7.17] ›Release notes

Elasticsearch version 7.5.0

edit

Elasticsearch version 7.5.0

edit

Also see Breaking changes in 7.5.

Known issues

edit
  • Applying deletes or updates on an index after it has been shrunk may corrupt the index. In order to prevent this issue, it is recommended to stop shrinking read-write indices. For read-only indices, it is recommended to force-merge indices after shrinking, which significantly reduces the likeliness of this corruption in the case that deletes/updates would be applied by mistake. This bug is fixed in Elasticsearch 7.7 and later versions. More details can be found on the corresponding issue.
  • Stop all transforms during a rolling upgrade to 7.5. If a transform is running during upgrade, the transform audit index might disappear. (issue: #49730)
  • Indices created in 6.x with date and date_nanos fields using formats that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results. https://github.com/elastic/elasticsearch/pull/52555 This is fixed in Elasticsearch 7.7 and later versions.

Breaking changes

edit
Search

Deprecations

edit
Aggregations
  • Deprecate the "index.max_adjacency_matrix_filters" setting #46394 (issue: #46324)
Allocation
  • Deprecate include_relocations setting #47443 (issue: #46079)
Mapping
Search

New features

edit
Features/ILM+SLM
Features/Ingest
Machine Learning
  • Implement evaluation API for multiclass classification problem #47126 (issue: #46735)
  • Implement new analysis type: classification #46537 (issue: #46735)
  • Add audit messages for Data Frame Analytics #46521 (issue: #184)
  • Implement DataFrameAnalyticsAuditMessage and DataFrameAnalyticsAuditor #45967
SQL

Enhancements

edit
Aggregations
  • Adjacency_matrix aggregation memory usage optimisation. #46257 (issue: #46212)
  • Support geotile_grid aggregation in composite agg sources #45810 (issue: #40568)
Allocation
  • Do not cancel ongoing recovery for noop copy on broken node #48265 (issue: #47974)
  • Shrink should not touch max_retries #47719
  • Re-fetch shard info of primary when new node joins #47035 (issues: #42518, #46959)
  • Sequence number based replica allocation #46959 (issue: #46318)
Authorization
  • Add support to retrieve all API keys if user has privilege #47274 (issue: #46887)
  • Add create_doc index privilege #45806
  • Reducing privileges needed by built-in beats_admin role #41586
CCR
  • Add Pause/Resume Auto-Follower APIs to High Level REST Client #47989 (issue: #47510)
  • Add Pause/Resume Auto Follower APIs #47510 (issue: #46665)
CRUD
  • Allow optype CREATE for append-only indexing operations #47169
Cluster Coordination
  • Warn on slow metadata persistence #47005
  • Improve LeaderCheck rejection messages #46998
Engine
  • Do not warm up searcher in engine constructor #48605 (issue: #47186)
  • Refresh should not acquire readLock #48414 (issue: #47186)
  • Avoid unneeded refresh with concurrent realtime gets #47895
  • sync before trimUnreferencedReaders to improve index preformance #47790 (issues: #46201, #46203)
  • Limit number of retaining translog files for peer recovery #47414
  • Remove isRecovering method from Engine #47039
Features/ILM+SLM
  • Separate SLM stop/start/status API from ILM #47710 (issue: #43663)
  • Set default SLM retention invocation time #47604 (issue: #43663)
  • ILM: Skip rolling indexes that are already rolled #47324 (issue: #44175)
  • Add support for POST requests to SLM Execute API #47061
  • Wait for snapshot completion in SLM snapshot invocation #47051 (issues: #38461, #43663)
  • Add node setting for disabling SLM #46794 (issue: #38461)
  • ILM: parse origination date from index name #46755 (issues: #42449, #46561)
  • [ILM] Add date setting to calculate index age #46561 (issue: #42449)
Features/Ingest
  • Add the ability to require an ingest pipeline #46847
Features/Java High Level REST Client
  • add function submitDeleteByQueryTask in class RestHighLevelClient #46833
  • return Cancellable in RestHighLevelClient #45688 (issue: #44802)
Features/Java Low Level REST Client
  • Add cloudId builder to the HLRC #47868
  • Add support for cancelling async requests in low-level REST client #45379 (issues: #43332, #44802)
Features/Monitoring
  • Remove hard coded version_created in default monitoring alerts #47744
Infra/Circuit Breakers
  • Emit log message when parent circuit breaker trips #47000
  • Fix G1 GC default IHOP #46169
Infra/Core
Infra/Packaging
  • Package the JDK into jdk.app on macOS #48765
  • Move ES_TMPDIR substitution into jvm options parser #47189 (issue: #47133)
  • Clarify missing java error message #46160 (issue: #44139)
Infra/Scripting
  • Add explanations to script score queries #46693
Infra/Settings
  • Do not reference values for filtered settings #48066
  • Allow setting validation against arbitrary types #47264 (issue: #25560)
  • Clarify error message on keystore write permissions #46321
  • Add more meaningful keystore version mismatch errors #46291 (issue: #44624)
Machine Learning
  • Throw an exception when memory usage estimation endpoint encounters empty data frame. #49143 (issue: #49140)
  • Change format of MulticlassConfusionMatrix result to be more self-explanatory #48174 (issue: #46735)
  • Make num_top_classes parameter’s default value equal to 2 #48119 (issue: #46735)
  • [ML] Add option to stop datafeed that finds no data #47922
  • Allow integer types for classification’s dependent variable #47902 (issue: #46735)
  • [ML] Add lazy assignment job config option #47726
  • [ML] Additional outlier detection parameters #47600
  • [ML] More accurate job memory overhead #47516
  • [ML] Throttle the delete-by-query of expired results #47177 (issues: #47003, #47103)
Mapping
Network
  • Introduce simple remote connection strategy #47480
  • Enhanced logging when transport is misconfigured to talk to HTTP port #45964 (issue: #32688)
Recovery
  • Do not send recovery requests with CancellableThreads #46287 (issue: #46178)
SQL
  • SQL: make date/datetime and interval types compatible in conditional functions #47595 (issue: #46674)
  • SQL: use calendar interval of 1y instead of fixed interval for grouping by YEAR and HISTOGRAMs #47558 (issue: #40162)
  • SQL: Support queries with HAVING over SELECT #46709 (issue: #37051)
  • SQL: Add support for shape type #46464 (issues: #43644, #46412)
Search
Snapshot/Restore
Transform
  • [ML][Transforms] allow executor to call start on started task #46347
  • [ML-DataFrame] improve error message for timeout case in stop #46131 (issue: #45610)
  • [ML][Data Frame] add support for wait_for_checkpoint flag on _stop API #45469 (issue: #45293)

Bug fixes

edit
Aggregations
  • Fix ignoring missing values in min/max aggregations #48970 (issue: #48905)
  • DocValueFormat implementation for date range fields #47472 (issues: #47323, #47469)
Allocation
  • Auto-expand replicated closed indices #48973
  • Handle negative free disk space in deciders #48392 (issue: #48380)
  • Dangling indices strip aliases #47581
  • Cancel recoveries even if all shards assigned #46520
  • Fail allocation of new primaries in empty cluster #43284 (issue: #41073)
Analysis
  • Reset Token position on reuse in predicate_token_filter #47424 (issue: #47197)
Audit
Authentication
  • Add owner flag parameter to the rest spec #48500 (issue: #48499)
  • Add populate_user_metadata in OIDC realm #48357 (issue: #48217)
  • Remove unnecessary details logged for OIDC #48271
  • Fix AD realm additional metadata #47179 (issue: #45848)
  • Fallback to realm authc if ApiKey fails #46538
  • PKI realm accept only verified certificates #45590
Authorization
  • Fix security origin for TokenService#findActiveTokensFor…​ #47418 (issue: #47151)
  • Use should clause instead of filter when querying native privileges #47019
  • Do not rewrite aliases on remove-index from aliases requests #46989
  • Validate index and cluster privilege names when creating a role #46361 (issue: #29703)
  • Validate query field when creating roles #46275 (issue: #34252)
CCR
  • CCR should auto-retry rejected execution exceptions #49213
  • Do not auto-follow closed indices #47721 (issue: #47582)
  • Relax maxSeqNoOfUpdates assertion in FollowingEngine #47188 (issue: #47137)
  • Handle lower retaining seqno retention lease error #46420 (issues: #46013, #46416)
CRUD
  • Close query cache on index service creation failure #48230 (issue: #48186)
  • Use optype CREATE for single auto-id index requests #47353
  • Ignore replication for noop updates #46458 (issues: #41065, #44603, #46366)
Client
  • Correct default refresh policy for security APIs #46896
Cluster Coordination
  • Ignore metadata of deleted indices at start #48918
  • Omit writing index metadata for non-replicated closed indices on data-only node #47285 (issue: #47276)
  • Assert no exceptions during state application #47090 (issue: #47038)
  • Remove trailing comma from nodes lists #46484
Distributed
Engine
  • Greedily advance safe commit on new global checkpoint #48559 (issue: #48532)
Features/ILM+SLM
  • Don’t halt policy execution on policy trigger exception #49128
  • Don’t schedule SLM jobs when services have been stopped #48658 (issue: #47749)
  • Ensure SLM stats does not block an in-place upgrade from 7.4 #48367
  • Ensure SLM stats does not block an in-place upgrade from 7.4 #48361
  • Add SLM support to xpack usage and info APIs #48096 (issue: #43663)
  • Change policy_id to list type in slm.get_lifecycle #47766 (issue: #47765)
  • Throw error retrieving non-existent SLM policy #47679 (issue: #47664)
  • Handle partial failure retrieving segments in SegmentCountStep #46556
  • Fixes for API specification #46522
Features/Indices APIs
  • Fix Rollover error when alias has closed indices #47148 (issue: #47146)
Features/Ingest
  • Do not wrap ingest processor exception with IAE #48816 (issue: #48810)
  • Introduce dedicated ingest processor exception #48810 (issue: #48803)
  • Allow dropping documents with auto-generated ID #46773 (issue: #46678)
  • Expose cache setting in UserAgentPlugin #46533
Features/Java High Level REST Client
  • fix incorrect comparison #48208
  • Fix ILM HLRC Javadoc→Documentation links #48083
  • Change HLRC count request to accept a QueryBuilder #46904 (issue: #46829)
  • [HLRC] Send min_score as query string parameter to the count API #46829 (issue: #46474)
  • HLRC multisearchTemplate forgot params #46492 (issue: #46488)
  • Added fields for MultiTermVectors (#42232) #42877 (issue: #42232)
Features/Java Low Level REST Client
Features/Monitoring
  • [Monitoring] Add new cluster privilege now necessary for the stack monitoring ui #47871
  • Validating monitoring hosts setting while parsing #47246 (issue: #47125)
Features/Watcher
  • Fix class used to initialize logger in Watcher #46467
  • Fix wrong URL encoding in watcher HTTP client #45894 (issue: #44970)
  • Prevent deadlock by using separate schedulers #48697 (issues: #41451, #47599)
  • Fix cluster alert for watcher/monitoring IndexOutOfBoundsExcep… #45308 (issue: #43184)
Geo
  • Geo: implement proper handling of out of bounds geo points #47734 (issue: #43916)
  • Geo: Fixes indexing of linestrings that go around the globe #47471 (issues: #43826, #43837)
  • Provide better error when updating geo_shape field mapper settings #47281 (issue: #47006)
  • Geo: fix indexing of west to east linestrings crossing the antimeridian #46601 (issue: #43775)
  • Reset queryGeometry in ShapeQueryTests #45974 (issue: #45628)
Highlighting
  • Fix highlighting of overlapping terms in the unified highlighter #47227
  • Fix highlighting for script_score query #46507 (issue: #46471)
Infra/Core
  • Don’t drop user’s MaxDirectMemorySize flag on jdk8/windows #48657 (issues: #44174, #48365)
  • Warn when MaxDirectMemorySize may be incorrect (Windows/JDK8 only issue) #48365 (issue: #47384)
  • Support optional parsers in any order with DateMathParser and roundup #46654 (issue: #45284)
Infra/Logging
  • SearchSlowLog uses a non thread-safe object to escape json #48363 (issues: #44642, #48358)
Infra/Scripting
  • Drop stored scripts with the old style-id #48078 (issue: #47593)
Machine Learning
  • [ML] Fixes for stop datafeed edge cases #49191 (issues: #43670, #48931)
  • [ML] Avoid NPE when node load is calculated on job assignment #49186 (issue: #49150)
  • Do not throw exceptions resulting from persisting datafeed timing stats. #49044 (issue: #49032)
  • [ML] Deduplicate multi-fields for data frame analytics #48799 (issues: #48756, #48770)
  • [ML] Prevent fetching multi-field from source #48770 (issue: #48756)
  • [ML] Fix detection of syslog-like timestamp in find_file_structure #47970
  • Fix serialization of evaluation response. #47557
  • [ML] Reinstate ML daily maintenance actions #47103 (issue: #47003)
  • [ML] fix two datafeed flush lockup bugs #46982
Network
  • Fix es.http.cname_in_publish_address Deprecation Logging #47451 (issue: #47436)
Recovery
  • Ignore Lucene index in peer recovery if translog corrupted #49114
Reindex
  • Fix issues with serializing BulkByScrollResponse #45357
SQL
  • SQL: Fix issue with mins & hours for DATEDIFF #49252
  • SQL: Fix issue with negative literels and parentheses #48113 (issue: #48009)
  • SQL: add "format" for "full" date range queries #48073 (issue: #48033)
  • SQL: Fix arg verification for DateAddProcessor #48041
  • SQL: Fix Nullability of DATEADD #47921
  • SQL: Allow whitespaces in escape patterns #47577 (issue: #47401)
  • SQL: fix multi full-text functions usage with aggregate functions #47444 (issue: #47365)
  • SQL: Check case where the pivot limit is reached #47121 (issue: #47002)
  • SQL: Properly handle indices with no/empty mapping #46775 (issue: #46757)
  • SQL: improve ResultSet behavior when no rows are available #46753 (issue: #46750)
  • SQL: use the correct data type for types conversion #46574 (issue: #46090)
  • SQL: Fix issue with common type resolution #46565 (issue: #46551)
  • SQL: fix scripting for grouped by datetime functions #46421 (issue: #40241)
  • SQL: Use null schema response #46386 (issue: #46381)
  • SQL: Fix issue with IIF function when condition folds #46290 (issue: #46268)
  • SQL: Fix issue with DataType for CASE with NULL #46173 (issue: #46032)
  • SQL: Failing Group By queries due to different ExpressionIds #43072 (issues: #33361, #34543, #36074, #37044, #40001, #40240, #41159, #42041, #46316)
  • SQL: wrong number of values for columns #42122
Search
  • Lucene#asSequentialBits gets the leadCost backwards. #48335
  • Ensure that we don’t call listener twice when detecting a partial failures in _search #47694
  • Fix alias field resolution in match query #47369
  • Multi-get requests should wait for search active #46283 (issue: #27500)
  • Resolve the incorrect scroll_current when delete or close index #45226
  • Don’t apply the plugin’s reader wrapper in can_match phase #47816 (issue: #46817)
Security
  • Remove uniqueness constraint for API key name and make it optional #47549 (issue: #46646)
  • Initialize document subset bit set cache used for DLS #46211 (issue: #45147)
Snapshot/Restore
  • Fix RepoCleanup not Removed on Master-Failover #49217
  • Make FsBlobContainer Listing Resilient to Concurrent Modifications #49142 (issue: #37581)
  • Fix SnapshotShardStatus Reporting for Failed Shard #48556 (issue: #48526)
  • Cleanup Concurrent RepositoryData Loading #48329 (issue: #48122)
  • Fix Bug in Azure Repo Exception Handling #47968
  • Make loadShardSnapshot Exceptions Consistent #47728 (issue: #47507)
  • Fix Snapshot Corruption in Edge Case #47552 (issues: #46250, #47550)
  • Fix Bug in Snapshot Status Response Timestamps #46919 (issue: #46913)
  • Normalize Blob Store Repo Paths #46869 (issue: #41814)
  • GCS deleteBlobsIgnoringIfNotExists should catch StorageException #46832 (issue: #46772)
  • Execute SnapshotsService Error Callback on Generic Thread #46277
  • Make Snapshot Logic Write Metadata after Segments #45689 (issue: #41581)
Store
  • Allow truncation of clean translog #47866
Task Management
  • Fix .tasks index strict mapping: parent_id should be parent_task_id #48393
Transform
  • [Transform] do not fail checkpoint creation due to global checkpoint mismatch #48423 (issue: #48379)
  • [7.5][Transform] prevent assignment if any node is older than 7.4 #48055 (issue: #48019)
  • [Transform] prevent assignment to nodes older than 7.4 #48044 (issue: #48019)
  • [ML][Transforms] fix bwc serialization with 7.3 #48021
  • [ML][Transforms] signal listener early on task _stop failure #47954
  • [ML][Transform] Use field_caps API for mapping deduction #46703 (issue: #46694)
  • [ML-DataFrame] Fix off-by-one error in checkpoint operations_behind #46235

Regressions

edit
Aggregations
  • Implement rounding optimization for fixed offset timezones #46670 (issue: #45702)
Infra/Core
  • [Java.time] Support partial parsing #46814 (issues: #45284, #47473)
  • Enable ResolverStyle.STRICT for java formatters #46675

Upgrades

edit
Infra/Scripting
  • Update mustache dependency to 0.9.6 #46243
Snapshot/Restore
  • Update AWS SDK for repository-s3 plugin to support IAM Roles for Service Accounts #46969
  • Upgrade to Azure SDK 8.4.0 #46094
Store
  • Upgrade to Lucene 8.3. #48829
« Elasticsearch version 7.5.1 Elasticsearch version 7.4.2 »