NOTE: You are looking at documentation for an older release. For the latest information, see the current release documentation.
Elasticsearch version 6.4.0
editElasticsearch version 6.4.0
editKnown issues
editElasticsearch 6.4.0 fails to start when PEM encoded private key files that have been exported from PKCS#12
keystores are in use. These files can be identified by the existence of lines that start with either
Bag Attributes
or Key Attributes
in the beginning of the key file.
These keys can be converted to a supported format using one of the following methods:
-
Remove the extra lines from the beginning of the file so that the file starts with the line that starts
with
-----BEGIN
-
Use openssl to convert it, i.e:
openssl pkcs8 -in old_format.key -topk8 -nocrypt -out new_format.key
assuming the key was not password protected.
Also see 6.4.
Breaking Changes
edit- Plugins
-
- Plugins: Remove meta plugins #30670
- Search
- Task Management
Breaking Java changes
editDeprecations
edit- Analysis
- Index APIs
- Java High Level REST Client
-
- Add high-level client methods that accept RequestOptions #31069
- Java Low Level REST Client
-
- Client: Deprecate many argument performRequest #30315
- Mapping
-
- Deprecate unindexed phrases #31072
- Scripting
- Search
-
- In the field capabilities API, deprecate support for providing fields in the request body. #30157
- Suggesters
New Features
edit- Aggregations
- Analysis
- Authentication
- Authorization
-
- Introduce Application Privileges with support for Kibana RBAC #32309
- Java High Level REST Client
- Machine learning
- Mapping
- Network
- Plugins
- SQL
- Scripting
-
- Add more contexts to painless execute api #30511
- Search
- Watcher
-
- Make watcher settings reloadable #31746
Enhancements
edit- Aggregations
- Analysis
- Audit
- Authentication
- CRUD
- Circuit Breakers
- Core
- Discovery-Plugins
- Distributed
-
- Avoid sending duplicate remote failed shard requests #31313
- Engine
- Geo
- Highlighting
-
- Bypass highlight query terms extraction on empty fields #32090
- Index APIs
- Ingest
-
- Extend KV Processor (#31789) #32232 (issue: #31786)
- Make a few Processors callable by Painless #32170
- date_index_name processor template resolution #31841
- Introduction of a bytes processor #31733
- Extend allowed characters for grok field names #31653, #31722 (issue: #21745)
- Ingest: Add ignore_missing option to RemoveProc #31693 (issues: #23086)
- Enable Templated Fieldnames in Rename #31690 (issue: #29657)
- Add region ISO code to GeoIP Ingest plugin #31669
- Extend allowed characters for grok field names #31653 (issue: #21745)
-
Add ingest-attachment support for per document
indexed_chars
limit #31352
- Java High Level REST Client
-
- Add Snapshots Status API to High Level Rest Client #32295, #31515
- Add put watch action #32026, #32191 (issue: #29827)
- Add Get Snapshots High Level REST API #31980
- Add X-Pack usage api #31975
- Check that client methods match API defined in the REST spec #31825
- Clean Up Snapshot Create Rest API #31779
- Add cluster get settings API #31706 (issue: #27205)
- Add get index API #31703 (issues: #27205)
- Turn GetFieldMappingsResponse to ToXContentObject #31544
- Add Get Snapshots High Level REST API #31537 (issue: #27205)
- Add Snapshots Status API to High Level Rest Client #31515 (issue: #27205)
- Add get field mappings to High Level REST API Client #31423 (issue: #27205)
- Add delete snapshot High Level REST API #31393 (issue: #27205)
- Add explain High Level REST API #31387 (issue: #27205)
- Add get stored script and delete stored script to high level REST API #31355 (issue: #27205)
- Add Create Snapshot to High-Level Rest Client #31215
- Add get index templates API #31161 (issue: #27205)
- Add simulate pipeline API #31158 (issue: #27205)
- Add validate query API #31077 (issue: #27205)
- Moved pipeline APIs to ingest namespace #31027
- List tasks failure to not lose nodeId #31001
- Add Verify Repository High Level REST API #30934 (issue: #27205)
- Move list tasks API under tasks namespace #30906 (issue: #29546)
- Add get mappings support to high-level rest client #30889 (issue: #27205)
-
Fix
AliasMetaData#fromXContent
parsing #30866 (issue: #28799) - Add delete ingest pipeline API #30865 (issues: #27205)
- Add get ingest pipeline API #30847 (issues: #27205)
- Add MultiSearchTemplate support to High Level Rest client #30836
- Add put ingest pipeline API #30793 (issue: #27205)
- Add cancel task API #30745 (issue: #27205)
- Add Delete Repository High Level REST API #30666 (issue: #27205)
- Add synced flush API #30650 (issues: #27205)
- Add PUT Repository High Level REST API #30501 (issue: #27205)
- Allow caller to set per request options #30490
- Add put index template api to high level rest client #30400 (issue: #27205)
- Add GET Repository High Level REST API #30362 (issue: #27205)
- Add support for field capabilities to the high-level REST client. #29664 (issue: #27205)
- Add Cluster Health API #29331 (issue: #27205)
- Add Get Settings API support to java high-level rest client #29229
- Add Get Aliases API to the high-level REST client #28799 (issue: #27205)
- Register ERR metric with NamedXContentRegistry #32320
- Java Low Level REST Client
-
- Node selector per client rather than per request #31471
- NodeSelector for node attributes #31296
- Replace Request#setHeaders with addHeader #30588
- Preserve REST client auth despite 401 response #30558
- Add String flavored setEntity #30447
- Refactor Sniffer and make it testable #29638 (issues: #25701, #27697)
- Add Request object flavored methods #29623
- License
- Logging
- Machine learning
-
- If a machine learning datafeed is configured to use cross cluster search to retrieve data, the remote clusters must have X-Pack installed and a valid licence for machine learning. If the licence requirements are not met, datafeeds using cross cluster search will not start. #31247
- Use default request durability for .ml-state index #32233
- Return statistics about forecasts as part of the job stats and usage API #31647 (issue: #31395)
- Add description to ML filters #31330
- Clean left behind model state docs #30659 (issue: #30551)
- Hide internal job update options from the REST API #30537
- Provide tmp storage for forecasting and possibly any machine learning native jobs #30399
- Improves and uses periodic boundary condition for seasonal component modeling (#84)
- Improves robustness with respect to outliers in detection and initialization of seasonal components (#90 (issue: #87))
- Improves behavior when there are abrupt changes in the seasonal components present in a time series (#91 (issue: #6))
- Adds explicit change point detection and modeling (#92)
- Improves partition analysis memory usage (#97)
- Reduces model memory by storing state for periodicity testing in a compressed format (#104, #100)
- Improves the accuracy of model memory control (#125, #122)
- Improves adaption of the modeling of cyclic components to very localized features (#138, #134)
- Reduces the memory consumed by distribution models (#162, #146)
- Forecasting of large machine learning jobs is now supported by temporarily storing model state on disk (#89)
- Secures the machine learning processes by preventing system calls such as fork and exec. The Linux implementation uses Seccomp BPF (secure computing with Berkeley Packet Filters) to intercept system calls and is available in kernels since 3.5. On Windows, Job Objects prevent new processes being created and macOS uses the sandbox functionality (#106, #98)
- Fixes a bug that caused underestimation of the memory used by shared pointers. Also reduces the memory consumed by unnecessary reference counting (#121, https://github.com/elastic/ml-cpp/pull/108, #115)
- Reduces model memory by storing the state for testing predictive calendar features in a compressed format (#137, #127)
- Always combine duplicate samples when updating population models (#74)
- Speeds up trend model component prediction (#73)
- Encodes distribution model weight style by offset in a fixed size weight array (#54)
- Mapping
- Monitoring
-
- _cluster/state should always return cluster_uuid #30143
- Network
-
- Backport SSL context names (#32223, #30953)
- Remove client connections from TcpTransport #31886 (issue: #31835)
- Support multiple system store types #31650
- Use remote client in TransportFieldCapsAction #30838
- Replace custom reloadable Key/TrustManager #30509
- Derive max composite buffers from max content len #29448
- Packaging
- Plugins
-
- Verify signatures on official plugins #30800
- Ranking
-
-
Rename ranking evaluation
quality_level
tometric_score
#32168 -
Rename ranking evaluation response
unknown_docs
section #32166 - Add Expected Reciprocal Rank metric #31891 (issue: #29653)
- Add details section for dcg ranking metric #31177
-
Move templated
_rank_eval
tests #30679 (issue: #30628) - Forbid expensive query parts in ranking evaluation #30151 (issue: #29674)
-
Rename ranking evaluation
- Rollup
-
-
Rollup now indexes
null
values, meaning a single "unified" job for heterogeneous data is now the recommended pattern. #31402 -
Rollup Search endpoint now supports the
terms
query. #30973) - Allow rollup job creation only if cluster is X-Pack ready. #30963
-
Rollups no longer allow patterns that match its
rollup_index
, which can lead to strange errors. #30491 - A new API allows getting the rollup capabilities of specific rollup indices, rather than by the target pattern. #30401
-
Validation errors thrown while creating a rollup job are now a specialization of the previous
ActionRequestValidationException
, which makes it easier to catch. The new exception isRollupActionRequestValidationException
. #30339 - Validate timezone in range queries to ensure they match the selected job when searching. #30338
-
Rollup now indexes
- SQL
-
- Allow LEFT and RIGHT as function names #32066 (issue: #32046)
- Add support for single parameter text manipulating functions #31874 (issue: #31604)
- Remove restriction for single column grouping #31818 (issue: #31793)
- Make a single JDBC driver jar #31012 (issue: #29856)
- Remove the last remaining server dependencies from JDBC #30771 (issue: #29856)
- Whitelist SQL utility class for better scripting #30681 (issue: #29832)
- Improve compatibility with MS query #30516 (issue: #30398)
- Reduce number of ranges generated for comparisons #30267 (issue: #30017)
- Teach the CLI to ignore empty commands #30265 (issue: #30000)
- JDBC driver prepared statement set* methods #31494 (issue: #31493)
- Scripting
- Search
-
- Force execution of fetch tasks #31974 (issue: #29442)
- Add second level of field collapsing #31808 (issue: #24855)
- Remove QueryCachingPolicy#ALWAYS_CACHE #31451
- Cross cluster search: don’t proxy requests for already connected node #31273
- Reject long regex in query_string #31136 (issue: #28344)
- Cross cluster search: do not use dedicated masters as gateways #30926 (issue: #30687)
- Added max_expansion param to span_multi #30913 (issue: #27432)
- Increase the maximum number of filters that may be in the cache. #30655
- Improve explanation in rescore #30629 (issue: #28725)
- Security
-
- Introduce fips_mode setting and associated checks #32326, #32344
- Tribe: Add error with secure settings copied to tribe #32298 (issue: #32117)
- Only auto-update license signature if all nodes ready #30859 (issues: #30731)
- Limit the scope of BouncyCastle dependency #30358
- Make licensing FIPS-140 compliant #30251
- Settings
- Snapshot/Restore
-
- ECS Task IAM profile credentials ignored in repository-s3 plugin #31864 (issues: #26913)
- Add write*Blob option to replace existing blob #31729
- Fixture for Minio testing #31688
- Do not check for object existence when deleting repository index files #31680
- Remove extra check for object existence in repository-gcs read object #31661 time an Azure object is accessed or modified #31617
- Lazy snapshot repository initialization #31606
- Do not check for S3 blob to exist before writing #31128
- Remove extra checks from HdfsBlobContainer #31126
- Allow date math for naming newly-created snapshots #30479 (issue: #7939 )
- Use simpler write-once semantics for HDFS repository #30439
- User proper write-once semantics for GCS repository #30438
- Use stronger write-once semantics for Azure repository #30437
- Use simpler write-once semantics for FS repository #30435
- Do not fail snapshot when deleting a missing snapshotted file #30332 (issue: #28322)
- Repository GCS plugin new client library #30168 (issue: #29259)
- Fail snapshot operations early on repository corruption #30140 (issues: #29649)
- Index name added to snapshot restore exception #29604 (issue: #27601)
- Do not load global state when deleting a snapshot #29278 (issue: #28934)
- Don’t load global state when only restoring indices #29239 (issue: #28934)
- Stats
- Store
-
-
Move caching of the size of a directory to
StoreDirectory
. #30581
-
Move caching of the size of a directory to
- Suggesters
- Task Management
- Transport API
- Watcher
-
- Clean up ensureWatchExists use #31926
- Store username on watch execution #31873 (issue: #31772)
- Consolidate setting update registration #31762
- Add secure setting for watcher email password #31620
- Slack message empty text #31596 (issue: #30071)
- Move watcher-history version setting to _meta field #30832 (issue: #30731)
- Only allow x-pack metadata if all nodes are ready #30743 (issues: #30731)
- Configure HttpClient parallel sent requests #30130
- Watcher: Make start/stop cycle more predictable and synchronous #30118
- ZenDiscovery
Bug Fixes
edit- Aggregations
-
- Fix profiling of ordered terms aggs #31814 (issue: #22123)
- Ensure that ip_range aggregations always return bucket keys. #30701 (issue: #21045)
- Fix class cast exception in BucketMetricsPipeline path traversal #30632 (issue: #30608)
- Fix NPE when CumulativeSum agg encounters null value/empty bucket #29641 (issue: #27544)
- Allocation
-
- A replica can be promoted and started in one cluster state update #32042
- Ignore numeric shard count if waiting for ALL #31265 (issue: #31151)
- Move allocation awareness attributes to list setting #30626 (issue: #30617)
- Auto-expand replicas when adding or removing nodes #30423 (issue: #1873)
- Auto-expand replicas only after failing nodes #30553
- Analysis
- Audit
-
- Fix audit index template upgrade loop #30779
- Authentication
-
- [Kerberos] Add debug log statement for exceptions #32663
- Remove Kerberos bootstrap checks #32451
- Fix building AD URL from domain name #31849
- resolveHasher defaults to NOOP #31723 (issues: #31697)
- Check auth scheme case insensitively #31490 (issue: #31486)
- Fix joining cluster with production license #31341 (issue: #31332)
- Fix token backwards compatibility with pre 6.0.0-beta2 #31254 (issues: #31195)
- Compliant SAML Response destination check #31175
- Clean up code in file stores #30348
- Fix TokenMetaData equals and hashcode #30347
- Authorization
-
- Fix role query that can match nested documents #32705
- Make get all application privileges require "*" permission #32460
- Revert to old way of merging automata #32254
- [PKI Realm] Invalidate cache on role mappings change #31510
- Fix dynamic mapping updates with aliases #30787 (issue: #30597)
- Include an empty JSON object in a JSON array when FLS filters out all fields #30709 (issue: #30624)
- Reduce garbage during index resolution #30180
- CRUD
-
- Bulk operation fail to replicate operations when a mapping update times out #30244
- Core
- Distributed
- Engine
- Geo
-
- Fix handling of points_only with term strategy in geo_shape #31766 (issue: #31707)
- Fix coerce validation_method in GeoBoundingBoxQueryBuilder #31747 (issue: #31718)
- Improve robustness of geo shape parser for malformed shapes #31449 (issue: #31428)
- Fix defaults in GeoShapeFieldMapper output #31302 (issue: #23206)
- Add support for indexed shape routing in geo_shape query #30760 (issue: #7663)
- Add validation that geohashes are not empty and don’t contain unsupported characters #30376 (issue: #23579)
- Index APIs
-
- Copy missing segment attributes in getSegmentInfo #32396
- Add support for is_write_index in put-alias body parsing #31674
- Fix writeIndex evaluation for aliases #31562
- Fix IndexTemplateMetaData parsing from xContent #30917
- Do not ignore request analysis/similarity settings on index resize operations when the source index already contains such settings. #30216
- Do not return all indices if a specific alias is requested via get aliases api. #29538 (issues: #27763)
- Ingest
- Java High Level REST Client
- Java Low Level REST Client
- License
-
- Do not serialize basic license expiration in X-Pack info #30848
- Machine learning
-
- Move open job failure explanation out of root cause #31925 (issue: #29950)
- Fix calendar and filter updates from non-master nodes #31804 (issue: #31803)
- Don’t treat stale FAILED jobs as OPENING in job allocation #31800 (issue: #31794)
- Rate limit established model memory updates #31768
- Account for gaps in data counts after job is reopened #30294 (issue: #30080)
- Ages seasonal components in proportion to the fraction of values with which they’re updated (#88 (issue: #87))
- Fixes persist and restore, which were missing some of the trend model state. (#103, #99)
- Stops zero variance data from generating a log error in the forecast confidence interval calculation (#120, #107)
- Fixes corner case which was failing to calculate lgamma values and fixes the corresponding log errors (#131, #126)
- Fixes influence count per bucket for metric population analyses, which was wrong and lead to incorrect influencer scoring (#153, #150)
- Fixes a possible SIGSEGV for jobs with multivariate by fields enabled, which caused the jobs to fail (#174, #170)
- Corrects the model bounds and typical value calculation for time series models which use a multimodal distribution. This issue could cause "Unable to bracket left percentile =…" errors to appear in the logs. (#178, #176)
- Mapping
-
- Make sure that field collapsing supports field aliases. #32648 (issue: #32623)
- Improve the error message when an index is incompatible with field aliases. #32482
- Make sure that field aliases count towards the total fields limit. #32222
-
Fix
range
queries on_type
field for singe type indices (#31756) #32161, #31756 (issues: #31476) - In NumberFieldType equals and hashCode, make sure that NumberType is taken into account. #31514
- Get Mapping API to honour allow_no_indices and ignore_unavailable #31507 (issue: #31485)
- Make sure KeywordFieldMapper#clone preserves split_queries_on_whitespace. #31049
- Delay _uid field data deprecation warning #30651 (issue: #30625)
- Monitoring
-
- Fix _cluster/state to always return cluster_uuid #30656
- Network
- Packaging
-
- Add temporary directory cleanup workarounds #32615 (issue: #31732)
- Add package pre-install check for java binary #31343 (issue: #29665)
-
Do not run
sysctl
forvm.max_map_count
when its already set #31285 - Stable filemode for zip distributions #30854 (issue: #30799)
- Force stable file modes for built packages #30823 (issue: #30799)
- Plugins
- REST API
- Recovery
-
- IndicesClusterStateService should replace an init. replica with an init. primary with the same aId #32374 (issue: #32308)
- Ensure to release translog snapshot in primary-replica resync #32045 (issue: #32030)
- Fix missing historyUUID in peer recovery when rolling upgrade 5.x to 6.3 #31506 (issue: #31482)
- Cancelling a peer recovery on the source can leak a primary permit #30318
- ReplicationTracker.markAllocationIdAsInSync may hang if allocation is cancelled #30316
- Do not log warn shard not-available exception in replication #30205
- Rollup
-
- Move to 128bit document IDs for Rollup. The old IDs were not wide enough and susceptible to hashing collisions. Jobs that are running during cluster upgrade will "self-upgrade" to the new ID scheme, but it is recommended that users fully rebuild Rollup indices from scratch if possible. Any existing collisions are not fixable and so data-loss may affect the rollup index despite the new IDs being used. #32558 (issue: #32372)
-
Histo group configurations should support
scaled_float
#32048 (issue: #32035) -
Fix rollup on date fields that don’t support
epoch_millis
#31890 - Metric config properly validates itself now #31159
- SQL
-
- HAVING clause should accept only aggregates #31872 (issue: #31726)
- Check timeZone argument in AbstractSqlQueryRequest #31822
- Fix incorrect HAVING equality #31820 (issue: #31796)
- Fix incorrect message for aliases #31792 (issue: #31611)
- Allow long literals #31777 (issue: #31750)
- Fix stackoverflow on getObject and timestamp conversion #31735 (issue: #31734)
- Fix rest endpoint names in node stats #31371
- Preserve scoring in bool queries #30730 (issue: #29685)
- Verify GROUP BY ordering on grouped columns #30585 (issue: #29900)
- SYS TABLES ordered according to *DBC specs #30530
- Fix parsing of dates with milliseconds #30419 (issue: #30002)
- Improve correctness of SYS COLUMNS & TYPES #30418 (issue: #30386)
- Fix bug caused by empty composites #30343 (issue: #30292)
- Correct error message #30138 (issue: #30016)
- Add BinaryMathProcessor to named writeables list #30127 (issue: #30014)
- Scripting
- Search
-
- Fix multi level nested sort #32204 (issues: #31554, #31783, #32130)
- Fix race in clear scroll #31259
- Fix index prefixes to work with span_multi #31066 (issue: #31056)
- Cross Cluster Search: preserve remote status code #30976 (issue: #27461)
- Fix NPE in more_like_this when field has zero tokens #30365 (issue: #30148)
- Fix failure for validate API on a terms query #30319, #29483 (issue: #29033)
- Fix a bug in FieldCapabilitiesRequest#equals and hashCode. #30181
- Fix TermsSetQueryBuilder.doEquals() method #29629 (issue: #29620)
- Add additional shards routing info in ShardSearchRequest #29533 (issue: #27550)
-
Use date format in
date_range
mapping before fallback to default #29310 (issue: #29282)
- Security
- Snapshot/Restore
- Store
- Suggesters
- Transport API
- Watcher
-
- Guard against null in email admin watches #32923 (issue: #32590)
- Fix null failure in watcher test #31968 (issue: #31948)
- Fix chain input toXcontent serialization #31721
- Add ssl.trust email account setting #31684
- Fix check for currently executed watches #31137
- Prevent duplicate watch triggering during upgrade #30643
- Prevent triggering watch when using activate API #30613
- Ensure trigger service pauses execution #30363
- Fix watch history template for dynamic slack attachments #30172
- Ensure mail message ids are unique per watch action #30112
- Validate xContentType in PutWatchRequest. #31088 (issue: #30057)
- ZenDiscovery