Elasticsearch version 7.15.0
editElasticsearch version 7.15.0
editAlso see Breaking changes in 7.15.
Known issues
edit-
Frozen tier: (Windows only) The frozen data tier relies on multiple caching mechanisms to speed up access to searchable snapshot files. One of these caches uses sparse files to avoid creating large files on disk when it is not strictly required. A bug prevented files from being created with the right options to enable sparse support on Windows, leading Elasticsearch to create potentially large files that can end up consuming all the disk space.
This issue is fixed in Elasticsearch versions 7.15.2 and later. There is no known workaround for earlier versions. Filesystems that enable sparse files by default are not affected. For more details, see #79371.
-
Aggregations: In Elasticsearch 7.14.0–7.15.0, when a cross-cluster search (CCS) request is proxied, the memory for the aggregations on the proxy node will not be freed. The trigger is cross-cluster search using aggregations where minimize roundtrips is not effective (for example, when minimize roundtrips is explicitly disabled, or implicitly disabled when using scroll, async and point-in-time searches).
This affects Kibana CCS aggregations because Kibana uses async search by default. This issue can also happen in all modes of remote connections configured for cross-cluster search (sniff and proxy). In sniff mode, we only connect to a subset of the remote nodes (by default 3). So if the remote node we want to send a request to is not one of those 3, we must send the request as a proxy request. The workaround is to periodically restart nodes with heap pressure.
We have fixed this issue in Elasticsearch 7.15.1 and later versions. For more details, see #78404.
Breaking changes
edit- Infra/Core
- Search
Deprecations
edit- Store
-
- Deprecate SimpleFS and replace it with NIOFS #75156
New features
editEnhancements
edit- Allocation
- Authentication
- Authorization
- Cluster Coordination
- Data streams
- EQL
-
- Minimise CCS roundtrips #76076
- Engine
- Geo
- ILM+SLM
- Infra/CLI
-
-
In
MultiCommand,
show command list when required #75993
-
In
- Infra/Logging
-
- Adds minimal traceparent header support to Elasticsearch #74210
- Infra/Plugins
- Infra/Resiliency
- Infra/Scripting
-
- Add Fields API to aggregation scripts and field scripts #76325
- Add ability to allow list instance methods on the script class #76045
- Add fields api to runtime fields contexts #76265
- Script: Def encoding parser #74840
- Script: Fields API for Filter context #76119
- Script: Fields API for Sort and Score scripts #75863 (issue: #61388)
- Script: Mangle user function names #72892 (issue: #69742)
- Script: ulong via fields API #76519 (issue: #64361)
- Ingest
-
-
Add support for
_meta
field to ingest pipelines #75905 -
Configurable
media_type
for mustache template encoding on append processor #76210 -
Json processor:
add_to_root_conflict_strategy
option #74967 - Json processor: allow duplicate keys #74956
-
Map iteration support for
ForEach
processor #64062 (issue: #55215) -
Support for wildcards and override option for
dot_expander
processor #74601
-
Add support for
- License
-
- Add persistent licensed feature tracking #76476
- Machine Learning
-
-
Add
datafeed_config
field to anomaly detection job configs #74265 -
Adding
datafeed_config
to job in high level rest client #75338 -
Adding new
p_value
scoring heuristic to significant terms aggregation #75313 - Delete expired annotations #75617 (issue: #75572)
- Deleting a job now deletes the datafeed if necessary #76010
- Integrating ML with the node shutdown API #75188
- Throttle job audit msgs if delayed data occurs for consecutive buckets #75815 (issue: #73679)
- Speed up training of regression and classification models on very large data sets #1941
- Improve regression and classification training accuracy for small data sets #1960
- Prune models for split fields (by, partition) that haven’t seen data updates for a given period of time #1962
-
Add
- Mapping
- Network
- Other
- Packaging
- Query Languages
-
-
Introduce
ParserUtils
to consolidate code #76399
-
Introduce
- Recovery
-
- Add peer recoveries using snapshot files when possible #76237 (issue: #73496)
- Add peer recovery planners that take into account available snapshots #75840
-
Compute latest snapshot directly in
TransportGetShardSnapshotAction
#76254 (issue: #73496) -
Keep track of data recovered from snapshots in
RecoveryState
#76499 (issue: #73496)
- Reindex
-
- Log reindex bulk request completion #75935
- Search
-
- Add index disk usage API #74051 (issue: #74051)
- Allow specifying index in pinned queries #74873 (issue: #67855)
- Compress async search responses before storing #74766 (issue: #67594)
- Emit multiple fields from a runtime field script #75108 (issue: #68203)
-
Replaced
field_masking_span
occurrences with respectiveParseField
#74718 (issue: #63527) - Search performance - better caching logic for queries on wildcard field #76035 (issue: #75848)
- Support search slicing with point-in-time #74457 (issue: #65740)
- Track Lucene field usage #74227 (issue: #73944)
- Unsigned long should accept value with zero fraction #75646 (issue: #67565)
- Security
-
- A new search API for API keys - core search function #75335 (issues: #71023, #73705)
-
Add sort and pagination support for
QueryApiKey
API #76144 (issue: #71023) - HLRC support for query API key API #76520 (issues: #71023, #75335, #76144)
- Query API key - Rest spec and yaml tests #76238 (issue: #71023)
- Return file-backed service tokens from all nodes #75200 (issue: #74530)
-
OnChange
listener now only runs when there is actual change #75207 (issue: #44701)
- Snapshot/Restore
-
- Add the ability to fetch the latest successful shard snapshot #75080 (issue: #73496)
-
Fix
SnapshotInfo#toXContent
delegation #75899 - Implement Numeric Offset Parameter in Get Snapshots API #76233 (issue: #74350)
- Include extra snapshot details in logs/APIs #75917
- Prevent searchable snapshots indices to be shrunk/split #75227 (issue: #56595)
- Return Total Result Count and Remaining Count in Get Snapshots Response #76150
- Stats
- Store
-
- Replace internal usages of SimpleFS with NIOFS #74996
- TLS
- Task Management
- Transform
-
-
Align transform checkpoint range with
date_histogram
interval for better performance #74004 (issue: #62746) - Changing hard coded 10k page size limit to 65k #74651 (issue: #57719)
- Improve performance by using point in time API for search #74984 (issue: #73481)
- Optimize composite agg execution using ordered groupings #75424
-
Align transform checkpoint range with
Bug fixes
edit- Aggregations
- Allocation
- Distributed
-
-
Wrap exception with
SendRequestTransportException
instead ofTransportException
#73551
-
Wrap exception with
- Geo
- ILM+SLM
-
-
Changing
wait_for_snapshot
to check start time rather than finish time #75644
-
Changing
- Indices APIs
-
- Fix pending deletes timeout log message #75776
- Infra/Core
- Infra/Scripting
-
- Fix the error message for instance methods that don’t exist #76512
- Ingest
- Machine Learning
-
- Allow for larger models in the inference step for data frame analytics #76116 (issue: #76093)
- Ignore exceptions while opening job after SIGTERM to JVM #75850
- Prevent accidentally asking for more resources when scaling down and improve scaling size estimations #74691 (issue: #74709)
- When setting upgrade mode ensure that internal actions don’t throw unnecessary permissions errors #75880
- Fix potential "process stopped unexpectedly: Fatal error" for training regression and classification models #1997 (issue: #1956)
- Network
- Search
- Security
- Snapshot/Restore
- Transform