IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Elasticsearch version 7.11.1 Elasticsearch version 7.10.2 »

› ›

Elasticsearch version 7.11.0

edit

Elasticsearch version 7.11.0

edit

Also see Breaking changes in 7.11.

Known issues

edit

Integration with Active Directory realms and LDAP realms is impacted by an issue that prevents Elasticsearch from starting. If you have configured an Active Directory or LDAP realm, then Elasticseach will fail to start with an error message indicating that Could not initialize class com.unboundid.util.Debug. This exception is fatal. If you encounter this during an upgrade, because Elasticsearch failed during node construction, you can safely downgrade to your previous version of Elasticsearch. Note: these downgrade instructions only apply if you encounter this specific error message; in general, downgrades are not safe. Elasticsearch 7.11.1 will contain a fix for this issue. For more details, see #68838.
If autoscaling is enabled for machine learning, the administrator of the cluster should increase the cluster setting xpack.ml.max_open_jobs to the maximum value of 512. This allows autoscaling to run reliably as it relies on assigning jobs only via memory. Having xpack.ml.max_open_jobs as a small number may cause autoscaling to behave unexpectedly.
If autoscaling is enabled and job assignment takes an inordinate amount of time, scaling sizes may increase much higher than required. Elasticsearch 7.13.0 contains a fix for this. For more details, see #72509
Snapshot and restore: If an index is deleted while the cluster is concurrently taking more than one snapshot then there is a risk that one of the snapshots may never complete and also that some shard data may be lost from the repository, causing future restore operations to fail. To mitigate this problem, set snapshot.max_concurrent_operations: 1 to prevent concurrent snapshot operations:
```
PUT _cluster/settings
{
  "persistent" : {
    "snapshot.max_concurrent_operations" : 1
  }
}
```
This issue is fixed in Elasticsearch versions 7.13.1 and later. It is not possible to repair a repository once it is affected by this issue, so you must restore the repository from a backup, or clear the repository by executing DELETE _snapshot/<repository name>/*, or move to a fresh repository. For more details, see #73456.

FIPS 140-2 compliance

edit

If you don’t run Elasticsearch in FIPS 140-2 mode, skip this section.

In some cases, such as when using the Bouncy Castle FIPS module in approved only mode, Elasticsearch may raise unexpected exceptions that prevent startup.

In Elasticsearch 7.11.0, we added a new PBKDF2-compatible hashing algorithm to fix this. For more information, see the Elasticsearch FIPS 140-2 documentation.

Breaking changes

edit

Machine Learning

Add new flag exclude_generated that removes generated fields in GET config APIs #63899 (issue: #63055)

Mapping

Distinguish between simple matches with and without the terms index #63945

Templates

Index templates with data stream enabled can no longer contain index aliases or component templates with index aliases. A bug in Elasticsearch 7.10 and 7.9 previously allowed this. #67886

Deprecations

edit

Features/CAT APIs

Deprecate the local parameter of /_cat/indices #62198 (issue: #60718)
Deprecate the local parameter of /_cat/shards #62197 (issue: #60718)

Features/Indices APIs

Deprecate _upgrade API (#47678) #50484 (issue: #47678)

Security

Deprecate certificate generation without a CA, and add self-signed option #64037 (issue: #61884)

New features

edit

Aggregations

Support global ordinals in top_metrics #64967 (issue: #64774)
Add doc_count field mapper #64503 (issue: #58339)

Audit

Introduce new audit record for security configuration changes via API #62916

Features/Ingest

URI parts ingest processor #65150 (issue: #57481)

Geo

Add geo_line aggregation #41612 (issue: #41649)

Search

Introduce async search status API #62947 (issue: #57537)

Transform

Implement latest function for Transform #65304 (issue: #65869)

Enhancements

edit

Aggregations

Add hard_bounds support for histogram field-based histograms #64246 (issue: #62124)
Add support for histogram fields to rate aggregation #63289 (issue: #62939)
Add supports for upper and lower values on boxplot based on the IQR value #63617 (issue: #60466)
Add value_count mode to rate agg #63687 (issue: #63575)
Allow mixing set-based and regexp-based include and exclude #63325 (issue: #62246)
Consider query when optimizing date rounding #63403
Speed up date_histogram without children #63643 (issue: #61467)
Speed up interval rounding #63245
Use global ordinals in Cardinality aggregation when it makes sense #62560

Allocation

Add data tiers preference for snapshot blob cache system index #64004 (issue: #63204)

Audit

Log a warning when current license level does not permit auditing #65498

Authentication

Adding API for generating SAML SP metadata #64517 (issue: #49018)
Adds realm name OIDC _security/oidc/prepare and _security/oidc/authenticate APIs responses #64966 (issue: #53161)
Introduce an additional hasher (PBKDF2_STRETCH) #65328
Returning tokenGroups attribute as SID string instead of byte array #63509 (issue: #61173)

Authorization

Optimize role bitset in document level security #64673

CCR

Clarify puts of non-existent settings on followers #64560

Client

Adding authentication information to access token create APIs #62490 (issue: #59685)

EQL

Add ? character to : operator #65545 (issue: #65536)
Add wildcard functionality to : operator #65188 (issue: #65154)
Introduce list declaration for : operator #65230 (issue: #64388)

Engine

Add index commit id to searcher #63963
Realtime get from in-memory segment when possible #64504

Features/Data streams

Add custom metadata support to data steams #63991 (issue: #59195)
Alow data streams to be marked hidden #63987 (issue: #63889)
Include date in data stream backing index names #65205 (issue: #65012)
Migrate aliased indices to data stream #61525 (issue: #61046)
REST endpoint and tests for data stream migration #64415 (issue: #61046)

Features/Features

Make stack.templates.enabled a dynamic setting #63764 (issues: #37861, #38560, #62835)

Features/ILM+SLM

Add validation of the SLM schedule frequency #64452 (issue: #55450)
Allow readonly in the hot phase for ILM policies #64381 (issues: #52073, #58289, #64008)
Allow shrink in the hot phase for ILM policies #64008 (issues: #52073, #56377)
Add support for the searchable_snapshot action in the hot phase #64883 (issue: #64656)
Validate policy and actions against current license #65412 (issue: #60208)
Make SLM Run Snapshot Deletes in Parallel #62284 (issue: #59655)
Use data stream for ILM history #64521

Features/Indices APIs

Add a template parameter to override auto_create_index value #61858 (issue: #20640)
Make legacy template noop updates not change cluster state #64493 (issues: #57662, #57851)
Mark component and composable index template APIs as stable #65013

Features/Ingest

Add copy_from parameter for set ingest processor #63540 (issues: #51046, #55682, #56985)
Add resource_name field to attachment ingest processor to aid file type recognition #64389
Fix IngestDocument.deepCopy to support sets #63067 (issue: #63066)

Features/Monitoring

Add action to decommission legacy monitoring cluster alerts #64373 (issue: #62668)

Features/Stats

Ignore system indices from mapping stats and analysis stats #65220
Update file extensions #62019

Geo

Enable geo_distance and geo_bounding_box queries on geo_shape field type #64224 (issue: #54628)
geo_point runtime field implementation #63164

Highlighting

Make runtime fields highlightable #65560

Infra/Core

Add deprecation check for Java version #64996 (issues: #40754, #40756, #41092)
Add infrastructure for managing system indices #65604 (issue: #61656)
Add stream serialisation for the OffsetTime values #63568 (issue: #63521)
Add validation in policy files for missing codebases #64841
Use different G1GC options for small heaps #59667

Infra/Plugins

Introduce licensed plugins #64850

Infra/Scripting

Move helper into painless #63439

Infra/Settings

Enhance byte-size setting validation #65363 (issue: #64428)

License

Adding a warning header when a license is about to expire #64948 (issue: #60562)

Machine Learning

Add new setting xpack.ml.use_auto_machine_memory_percent for auto calculating native memory percentage allowed for jobs #63887 (issue: #63795)
Add new snapshot upgrader API for upgrading older snapshots #64665 (issue: #64154)
Add search runtime_mappings to datafeed configuration #65606
Add total ML memory to ML info #65195 (issue: #64225)
Adding result_type and mlcategory fields to category definitions #63326 (issue: #60108)
Increase log level for forecast disk storage problems #64766 (issue: #58806)
Provide a way to revert an anomaly detection job to an empty snapshot #65431
During regression and classification training prefer smaller models if performance is similar #1516
Add a response mechanism for commands sent to the native controller #1520, #63542 (issue: #62823)
Speed up anomaly detection for seasonal data. This is particularly effective for jobs using longer bucket lengths #1549
Fix an edge case which could cause typical and model plot bounds to blow up to around max double #1551
Estimate upper bound of potential gains before splitting a decision tree node to avoid unnecessary computation #1537
Improvements to time series modeling particularly in relation to adaption to change #1614
Warn and error log throttling #1615
Soften the effect of fluctuations in anomaly detection job memory usage on node assignment and add assignment_memory_basis to model_size_stats #1623, #65561 (issue: #63163)

Mapping

Add xpack info and usage endpoints for runtime fields #65600 (issue: #59332)
Enable collapse on unsigned_long field #63495 (issue: #60050)
Ensure field types consistency on supporting text queries #63487
Introduce dynamic runtime setting #65489
Introduce runtime section in mappings #62906 (issue: #59332)
Improve error message in case of invalid dynamic templates #60870 (issue: #51233)
Support unmapped fields in search fields option #65386 (issue: #63690)

Network

Include trusted issuer details in SSL diagnostics #61702
Remote/info should fail on node without remote_cluster_client role #65289

Packaging

Autodetermine heap settings based on node roles and total system memory #65905
Clarify usage of jvm.options in its comments #61456 (issue: #51882)

Query Languages

Combine multiple equal queries into IN #65353 (issues: #46477, #62804)
Improve expression normalization #65419 (issue: #65353)

SQL

Add method args to PERCENTILE / PERCENTILE_RANK #65026 (issue: #63567)
Enhance IN operator support for formatted date fields. Extend allowed conversions #63483 (issue: #58932)

Search

Add runtime_mappings to search request #64374 (issue: #59332)
Add default scripts for runtime geo-point fields #65586
Add support for runtime fields #61776 (issue: #59332)
Apply can match phase on coordinator when the min max field data is available at the coordinator #65583 (issue: #65689)
Cancel search on shard failure when partial results disallowed #63520 (issue: #60278)
Do not skip not available shard exception in search response #64337 (issue: #47700)

Snapshot/Restore

Add CacheFile#fsync() method to ensure cached data are written on disk #64201
Allow searchable snapshot cache service to periodically fsync cache files #64696 (issue: #64201)
Allow adding existing cache files in searchable snapshots cache service #65538
Allow creating a SparseFileTracker with ranges already present #65501
Expose the list of completed ranges in SparseFileTracker #64145
Limit the number of snapshots in a BlobStoreRepository #64461
Record timestamp field range in index metadata #65564
Upgrade Azure repository SDK to v12 #65140
Wait for prewarm when relocating searchable snapshot shards #65531

Task Management

Send ban parent per outstanding child connection #65443

Transform

Add new exclude_generated flag to GET transform #63093 (issue: #63055)
Add support for "missing" aggregation #63651
Add support for median absolute deviation #64634
Improve continuous transform date_histogram group_by with ingest timestamps #63315 (issues: #54254, #59061)
Use ISO dates in output instead of epoch millis #65584 (issue: #63787)

Bug fixes

edit

Aggregations

Prevent some BigArray leaking #64744
Return -/+ Infinity when min/max bounds are set null #64091
Return an error when a rate aggregation cannot calculate bucket sizes #65429 (issue: #63703)

Authentication

Don’t invalidate refresh token a second time #64259 (issue: #56903)
Fix a NPE bug caused by code keep executing after listener returns #64762 (issue: #64692)

Cluster Coordination

Simplify ClusterStateUpdateTask timeout handling #64117 (issue: #64116)

Features/Features

Some template alias settings are not persisted on PUT template #63162

Features/Indices APIs

Fix remove alias with must_exist #65141 (issues: #58100, #62642)
Preserve backing index ordering for data streams #63749 (issue: #61046)

Features/Ingest

Fail with a better error when providing null as processor config #64565 (issue: #57793)

Features/Java Low Level REST Client

Fix decompressed response headers #63419 (issue: #63314)

Geo

Compute polygon orientation using signed area #64095 (issue: #26286)
Fixes GeoLineAggregator bugs #65521 (issue: #65473)
Handle degenerated rectangles while indexing #64122

Infra/Core

Fix array handling in XContentMapValues.extractRawValues() #65193

Machine Learning

Add ignore_above for message.raw field in notifications index mappings #64455 (issue: #63888)
Fix edge case for data frame analytics where a field mapped as a keyword actually has boolean and string values in the _source #64826
Fix job ID in C++ logs for normalize and memory estimation #63874 (issues: #54636, #60395)
Truncate long audit messages #64849 (issue: #64570)
Fix potential cause for log errors from CXMeansOnline1d #1586
Fix scaling of some hyperparameters for Bayesian optimization #1612
Fix missing state in persist and restore for anomaly detection. This caused suboptimal modeling after a job was closed and reopened or failed over to a different node #1668

Mapping

Count only mapped fields towards docvalue_fields limit #63806 (issue: #63730)
Fix exception when building exists query for nested search-as-you-type field #64630 (issue: #64609)

Packaging

Fix plugin installation in Docker container as non-default user #63807
Rework Dockerfile instructions to save space #65308 (issue: #64274)

Search

Add explanation to runtime field query #63429
Correctly handle mixed object paths in XContentMapValues #65539 (issue: #65499)
Drop boost from runtime distance feature query #63949 (issue: #63767)
Handle range query edge case #63397 (issue: #63386)
Return partial failures if search was cancelled #64382 (issues: #63520, #64367)
_mtermvectors should continue request on item error #65324 (issue: #65228)

Security

Gracefully handle exceptions from security providers #65464

Snapshot/Restore

Fix inconsistent API behavior when cluster never had a snapshot repository configured #65535 (issue: #65511)
Fix two snapshot clone state machine bugs #65042
Correct the number of searchable snapshot prewarmers #63919 (issue: #59313)
Stricter check for snapshot restore version compatibility #65580 (issue: #65567)

Transform

Improve bucket key normalization #64196 (issue: #64070)
Make state handling more robust when stop is called while indexer shuts down #64922 (issue: #62204)
Weighted average should map to double #64586

« Elasticsearch version 7.11.1 Elasticsearch version 7.10.2 »