Elasticsearch version 6.2.0

edit

Elasticsearch version 6.2.0

edit

Breaking Changes

edit
Aggregations
  • Add a new cluster setting to limit the total number of buckets returned by a request #27581 (issues: #26012, #27452)
Core
  • Forbid granting the all permission in production #27548
Highlighting
  • Limit the analyzed text for highlighting #27934 (issue: #27517)
Rollover
  • Fail rollover if duplicated alias found in templates #28110 (issue: #26976)
Search
  • Introduce limit to the number of terms in Terms Query #27968 (issue: #18829)

Breaking Java Changes

edit
Java API
  • Remove operationThreaded from Java API #27836
Java High Level REST Client
  • REST high-level client: remove index suffix from indices client method names #28263

Deprecations

edit
Analysis
  • Backport delimited payload filter renaming #27535 (issue: #26625)
Suggesters
  • deprecating jarowinkler in favor of jaro_winkler #27526
  • Deprecating levenstein in favor of levensHtein #27409 (issue: #27325)

New Features

edit
Machine Learning
  • Added the ability to identify scheduled events and prevent anomaly detection during these periods. For more information, see Calendars and Scheduled Events.
Plugin Ingest GeoIp
  • Enable ASN support for Ingest GeoIP plugin. #27958 (issue: #27849)
Plugin Lang Painless
  • Painless: Add spi jar that will be published for extending whitelists #28302
  • Painless: Add a simple cache for whitelist methods and fields. #28142
Plugins
  • Add the ability to bundle multiple plugins into a meta plugin #28022 (issue: #27316)
Rank Evaluation
  • Backport of ranking evaluation API (#27478) #27844 (issue: #27478)
Recovery
  • Backport for using lastSyncedGlobalCheckpoint in deletion policy #27866 (issue: #27826)
Reindex API
  • Add scroll parameter to _reindex API #28041 (issue: #27555)
Security
  • X-Pack security now supports user authentication using SAML Single Sign on. For more information, see SAML authentication.
Watcher

Enhancements

edit
Allocation
  • Fix cluster.routing.allocation.enable and cluster.routing.rebalance.enable case #28037 (issue: #28007)
  • Add node id to shard failure message #28024 (issue: #28018)
Analysis
  • Limit the analyzed text for highlighting (#27934) #28176 (issue: #27517)
  • Allow TrimFilter to be used in custom normalizers #27758 (issue: #27310)
Circuit Breakers
  • Add accounting circuit breaker and track segment memory usage #27116 (issue: #27044)
Cluster
  • Adds wait_for_no_initializing_shards to cluster health API #27489 (issue: #25623)
Core
Discovery
  • Add information when master node left to DiscoveryNodes' shortSummary() #28197 (issue: #28169)
Engine
  • Move uid lock into LiveVersionMap #27905
  • Optimize version map for append-only indexing #27752
Geo
  • [GEO] Add WKT Support to GeoBoundingBoxQueryBuilder #27692 (issues: #27690, #9120)
  • [Geo] Add Well Known Text (WKT) Parsing Support to ShapeBuilders #27417 (issue: #9120)
Highlighting
  • Include all sentences smaller than fragment_size in the unified highlighter #28132 (issue: #28089)
Ingest
Internal
  • Make KeyedLock reentrant #27920
  • Make AbstractQueryBuilder.declareStandardFields to be protected (#27865) #27894 (issue: #27865)
  • Tighten the CountedBitSet class #27632
  • Avoid doing redundant work when checking for self references. #26927 (issue: #26907)
Java API
  • Add missing delegate methods to NodeIndicesStats #28092
  • Java api clean-up : consistency for shards_acknowledged getters #27819 (issue: #27784)
Java High Level REST Client
  • add toString implementation for UpdateRequest. #27997 (issue: #27986)
  • Add Close Index API to the high level REST client #27734 (issue: #27205)
  • Add Open Index API to the high level REST client #27574 (issue: #27205)
  • Added Create Index support to high-level REST client #27351 (issue: #27205)
  • Add multi get api to the high level rest client #27337 (issue: #27205)
  • Add msearch api to high level client #27274
Machine Learning
  • Increased tokenization flexibility for categorization. Now all Elasticsearch analyzer functionality is available, which opens up the possibility of sensibly categorizing non-English log messages. For more information, see Customizing the Categorization Analyzer.
  • Improved the sensitivity of the analysis to high variance data with lots of values near zero.
  • Improved the decay rate of the model memory by using a weighted moving average.
  • Machine learning indices created after upgrading to 6.2 have the auto_expand_replicas: 0-1 setting rather than a fixed setting of 1 replica. As a result, machine learning indices created after upgrading to 6.2 can have a green status on single node clusters. There is no impact in multi-node clusters.
  • Changed the credentials that are used by datafeeds. When X-Pack security is enabled, a datafeed stores the roles of the user who created or updated the datafeed at that time. This means that if those roles are updated, the datafeed subsequently runs with the new permissions that are associated with the roles. However, if the user’s roles are adjusted after creating or updating the datafeed then the datafeed continues to run with the permissions that are associated with the original roles. For more information, see Datafeeds.
  • Added a new scheduled forecast status, which indicates that the forecast has not started yet.
Mapping
Monitoring
  • X-Pack monitoring indices (.monitoring) created after upgrading to 6.2 have the auto_expand_replicas: 0-1 setting rather than a fixed setting of 1 replica. As a result, monitoring indices created after upgrading to 6.2 can have a green status on single node clusters. There is no impact in multi-node clusters.
  • Added a cluster alert that triggers whenever a node is added, removed, or restarted.
Network
  • Add NioGroup for use in different transports #27737 (issue: #27260)
  • Add read timeouts to http module #27713
  • Implement byte array reusage in NioTransport #27696 (issue: #27563)
  • Introduce resizable inbound byte buffer #27551 (issue: #27563)
  • Decouple nio constructs from the tcp transport #27484 (issue: #27260)
Packaging
Percolator
  • also extract match_all queries when indexing percolator queries #27585
Plugin Lang Painless
  • Painless: Add whitelist extensions #28161
  • Painless: Modify Loader to Load Classes Directly from Definition #28088
  • Clean Up Painless Cast Object #27794
  • Painless: Only allow Painless type names to be the same as the equivalent Java class. #27264
Plugins
  • Add client actions to action plugin #28280 (issue: #27759)
  • Plugins: Add validation to plugin descriptor parsing #27951
  • Plugins: Add plugin extension capabilities #27881
  • Add support for filtering mappings fields #27603
Rank Evaluation
  • Simplify RankEvalResponse output #28266
Recovery
  • Truncate tlog cli should assign global checkpoint #28192 (issue: #28181)
  • Replica starts peer recovery with safe commit #28181 (issue: #10708)
  • Primary send safe commit in file-based recovery #28038 (issue: #10708)
  • Fail resync-failed shards in subsequent writes #28005
  • Introduce promoting index shard state #28004 (issue: #24841)
  • Non-peer recovery should set the global checkpoint #27965
  • Persist global checkpoint when finalizing a peer recovery #27947 (issue: #27861)
  • Rollback a primary before recovering from translog #27804 (issue: #10708)
Search
  • Use typeName() to check field type in GeoShapeQueryBuilder #27730
  • Optimize search_after when sorting in index sort order #26401
Security
  • Added the ability to refresh tokens that were created by the token API. The API provides information about a refresh token, which you can use within 24 hours of its creation to extend the life of a token. For more information, see Token management APIs.
  • Added principal and role information to access_granted, access_denied, run_as_granted, and run_as_denied audit events. For more information about these events, see Auditing Security Events.
  • Added audit event ignore policies, which are a way to tune the verbosity of an audit trail. These policies define rules for ignoring audit events that match specific attribute values. For more information, see Logfile Audit Events Ignore Policies.
  • Added a certificates API, which enables you to retrieve information about the X.509 certificates that are used to encrypt communications in your Elasticsearch cluster. For more information, see SSL Certificate API.
Sequence IDs
  • Do not keep 5.x commits when having 6.x commits #28188 (issues: #27606, #28038)
  • Use lastSyncedGlobalCheckpoint in deletion policy #27826 (issue: #27606)
  • Use CountedBitSet in LocalCheckpointTracker #27793
  • Only fsync global checkpoint if needed #27652
  • Keep commits and translog up to the global checkpoint #27606
  • Adjust CombinedDeletionPolicy for multiple commits #27456 (issues: #10708, #27367)
  • Keeps index commits up to the current global checkpoint #27367 (issue: #10708)
  • Dedup translog operations by reading in reverse #27268 (issue: #10708)
Settings
  • Add validation of keystore setting names #27626
Snapshot/Restore
  • Use AmazonS3.doesObjectExist() method in S3BlobContainer #27723
  • Remove XContentType auto detection in BlobStoreRepository #27480
  • Include include_global_state in Snapshot status API (#22423) #26853 (issue: #22423)
Task Manager
  • Add ability to associate an ID with tasks #27764 (issue: #23250)
Translog
Watcher
  • Added the ability to set the index and doc_type dynamically in an index action. For more information, see Index Action.
  • Added a refresh index action attribute, which enables you to set the refresh policy of the write request. For more information, see Index Action.
  • Added support for actions in slack attachments, which enables you to add buttons that can be clicked in slack messages. For more information, see Slack Action.
  • Watcher indices (.watch* and triggered_watches) created after upgrading to 6.2 have the auto_expand_replicas: 0-1 setting rather than a fixed setting of 1 replica. As a result, Watcher indices created after upgrading to 6.2 can have a green status on single node clusters. There is no impact in multi-node clusters.

Bug Fixes

edit
Aggregations
  • Adds metadata to rewritten aggregations #28185 (issue: #28170)
  • Fix NPE on composite aggregation with sub-aggregations that need scores #28129
  • StringTerms.Bucket.getKeyAsNumber detection type #28118 (issue: #28012)
  • Fix incorrect results for aggregations nested under a nested aggregation #27946 (issue: #27912)
  • Fix global aggregation that requires breadth first and scores #27942 (issues: #22321, #27928)
  • Fix composite aggregation when after term is missing in the shard #27936
  • Fix preserving FiltersAggregationBuilder#keyed field on rewrite #27900 (issue: #27841)
  • Using DocValueFormat::parseBytesRef for parsing missing value parameter #27855 (issue: #27788)
  • Fix illegal cast of the "low cardinality" optimization of the terms aggregation. #27543
  • Always include the _index and _id for nested search hits. #27201 (issue: #27053)
Allocation
  • Do not open indices with broken settings #26995
Core
  • Fix lock accounting in releasable lock #28202
  • Fixes ByteSizeValue to serialise correctly #27702 (issue: #27568)
  • Do not set data paths on no local storage required #27587 (issue: #27572)
  • Ensure threadcontext is preserved when refresh listeners are invoked #27565
  • Ensure logging is configured for CLI commands #27523 (issue: #27521)
Engine
  • Replica recovery could go into an endless flushing loop #28350
  • Use _refresh to shrink the version map on inactivity #27918 (issue: #27852)
  • Allow resize version map under lock even if there are pending operations #27870 (issue: #27852)
  • Reset LiveVersionMap on sync commit #27534 (issue: #27516)
Geo
  • Correct two equality checks on incomparable types #27688
  • Handle case where the hole vertex is south of the containing polygon(s) #27685 (issue: #25933)
Highlighting
  • Fix highlighting on a keyword field that defines a normalizer #27604
Inner Hits
  • Add version support for inner hits in field collapsing (#27822) #27833 (issue: #27822)
Internal
  • Never return null from Strings.tokenizeToStringArray #28224 (issue: #28213)
  • Fallback to TransportMasterNodeAction for cluster health retries #28195 (issue: #28169)
  • Retain originalIndex info when rewriting FieldCapabilities requests #27761
Java REST Client
  • Do not use system properties when building the HttpAsyncClient #27829 (issue: #27827)
Machine Learning
  • Improved error reporting for crashes and resource problems on Linux.
  • Improved the detection of seasonal trends in bucket spans longer than 1 hour.
  • Updated the forecast API to wait for validation and return an error if the validation fails.
  • Set the actual bucket value to 0 in model plots for empty buckets for count and sum functions. The count and sum functions treat empty buckets as 0 rather than unknown for anomaly detection, so it was inconsistent not to do the same for model plots. This inconsistency resulted in problems plotting these buckets in Kibana.
Mapping
  • Ignore null value for range field (#27845) #28116 (issue: #27845)
  • Pass java.locale.providers=COMPAT to Java 9 onwards #28080 (issue: #10984)
  • Allow update of eager_global_ordinals on _parent. #28014 (issue: #24407)
  • Fix merging of _meta field #27352 (issue: #27323)
Network
  • Only bind loopback addresses when binding to local #28029 (issue: #1877)
  • Remove potential nio selector leak #27825
  • Fix issue where the incorrect buffers are written #27695 (issue: #27551)
  • Throw UOE from compressible bytes stream reset #27564 (issue: #24927)
  • Bubble exceptions when closing compressible streams #27542 (issue: #27540)
Packaging
  • Allow custom service names when installing on windows #25255 (issue: #25231)
Percolator
  • Avoid TooManyClauses exception if number of terms / ranges is exactly equal to 1024 #27519 (issue: #1)
Plugin Analysis ICU
  • Catch InvalidPathException in IcuCollationTokenFilterFactory #27202
Plugin Analysis Phonetic
  • Fix daitch_mokotoff phonetic filter to use the dedicated Lucene filter #28225 (issue: #28211)
Plugin Lang Painless
  • Painless: Fix variable scoping issue in lambdas #27571 (issue: #26760)
  • Painless: Fix errors allowing void to be assigned to def. #27460 (issue: #27210)
Plugin Repository HDFS
  • Fix SecurityException when HDFS Repository used against HA Namenodes #27196
Plugins
  • Make sure that we don’t detect files as maven coordinate when installing a plugin #28163
  • Fix upgrading indices which use a custom similarity plugin. #26985 (issue: #25350)
Recovery
Rollover
Scripting
  • Ensure we protect Collections obtained from scripts from self-referencing #28335
Scroll
  • Reject scroll query if size is 0 (#22552) #27842 (issue: #22552)
  • Fix scroll query with a sort that is a prefix of the index sort #27498
Search
  • Fix simple_query_string on invalid input #28219 (issue: #28204)
  • Use the underlying connection version for CCS connections #28093
  • Fix synonym phrase query expansion for cross_fields parsing #28045
  • Carry forward weights, etc on rescore rewrite #27981 (issue: #27979)
  • Fix routing with leading or trailing whitespace #27712 (issue: #27708)
Security
  • Updated the setup-passwords command to generate passwords with characters A-Z, a-z, and 0-9, so that they are safe to use in shell scripts. For more information about this command, see elasticsearch-setup-passwords.
  • Improved the error messages that occur if the x-pack directory is missing when you run users commands.
  • Fixed the ordering of realms in a realm chain, which determines the order in which the realms are consulted. For more information, see Realms.
Sequence IDs
  • Recovery from snapshot may leave seq# gaps #27850
  • No longer unidle shard during recovery #27757 (issue: #26591)
  • Obey translog durability in global checkpoint sync #27641
Settings
  • Settings: Introduce settings updater for a list of settings #28338 (issue: #28047)
  • Fix setting notification for complex setting (affixMap settings) that could cause transient settings to be ignored #28317 (issue: #28316)
  • Fix environment variable substitutions in list setting #28106 (issue: #27926)
  • Allow index settings to be reset by wildcards #27671 (issue: #27537)
Snapshot/Restore
  • Consistent updates of IndexShardSnapshotStatus #28130 (issue: #26480)
  • Avoid concurrent snapshot finalizations when deleting an INIT snapshot #28078 (issues: #27214, #27931, #27974)
  • Do not start snapshots that are deleted during initialization #27931
  • Do not swallow exception in ChecksumBlobStoreFormat.writeAtomic() #27597
  • Consistent update of stage and failure message in IndexShardSnapshotStatus #27557 (issue: #26480)
  • Fail restore when the shard allocations max retries count is reached #27493 (issue: #26865)
  • Delete shard store files before restoring a snapshot #27476 (issues: #20220, #26865)
Stats
  • Fixes DocStats to properly deal with shards that report -1 index size #27863
  • Include internal refreshes in refresh stats #27615
Term Vectors
  • Fix term vectors generator with keyword and normalizer #27608 (issue: #27320)
Watcher

Upgrades

edit
Core
  • Dependencies: Update joda time to 2.9.9 #28261
  • upgrade to lucene 7.2.1 #28218 (issue: #28044)
  • Upgrade jna from 4.4.0-1 to 4.5.1 #28183 (issue: #28172)
Ingest
  • update ingest-attachment to use Tika 1.17 and newer deps #27824