Breaking changes in 7.0

This section discusses the changes that you need to be aware of when migrating your application to Elasticsearch 7.0.

See also What’s new in 7.7 and Release notes.

Elasticsearch 7.0 can read indices created in version 6.0 or above. An Elasticsearch 7.0 node will not start in the presence of indices created in a version of Elasticsearch before 6.0.

These execution_hint are removed and should be replaced by global_ordinals.

The dynamic cluster setting named search.max_buckets now defaults to 10,000 (instead of unlimited in the previous version). Requests that try to return more than the limit will fail with an exception.

The missing option of the composite aggregation, deprecated in 6.x, has been removed. missing_bucket should be used instead.

The object used to share aggregation state between the scripts in a Scripted Metric Aggregation is now a variable called state available in the script context, rather than being provided via the params object as params._agg.

The metric aggregation has been changed to require these two script parameters to ensure users are explicitly defining how their data is processed.

The percentiles and percentile_ranks aggregations used to return NaN in the response if they were applied to an empty set of values. Because NaN is not officially supported by JSON, it has been replaced with null.

When the stats and extended_stats aggregations collected zero docs (doc_count: 0), their value would be null. This was in contrast with the sum aggregation which would return 0. The stats and extended_stats aggs are now consistent with sum and also return zero.

To safeguard against out of memory errors, the number of tokens that can be produced using the _analyze endpoint has been limited to 10000. This default limit can be changed for a particular index with the index setting index.analyze.max_token_count.

Highlighting a text that was indexed without offsets or term vectors, requires analysis of this text in memory real time during the search request. For large texts this analysis may take substantial amount of time and memory. To protect against this, the maximum number of characters that will be analyzed has been limited to 1000000. This default limit can be changed for a particular index with the index setting index.highlight.max_analyzed_offset.

The delimited_payload_filter was deprecated and renamed to delimited_payload in 6.2. Using it in indices created before 7.0 will issue deprecation warnings. Using the old name in new indices created in 7.0 will throw an error. Use the new name delimited_payload instead.

The standard token filter has been removed because it doesn’t change anything in the stream.

The standard_html_strip analyzer has been deprecated, and should be replaced with a combination of the standard tokenizer and html_strip char_filter. Indexes created using this analyzer will still be readable in elasticsearch 7.0, but it will not be possible to create new indexes using it.

The nGram and edgeNGram token filter names have been deprecated in an earlier 6.x version. Indexes created using these token filters will still be readable in elasticsearch 7.0 but indexing documents using those filter names will issue a deprecation warning. Using the deprecated names on new indices starting with version 7.0.0 will be prohibited and throw an error when indexing or analyzing documents. Both names should be replaced by ngram or edge_ngram respectively.

Due to cross-cluster search using : to separate a cluster and index name, cluster names may no longer contain :.

The default value for the wait_for_active_shards parameter of the open index API is changed from 0 to 1, which means that the command will now by default wait for all primary shards of the opened index to be allocated.

These shard preferences are removed in favour of the _prefer_nodes and _only_nodes preferences.

Clusters now have soft limits on the total number of open shards in the cluster based on the number of nodes and the cluster.max_shards_per_node cluster setting, to prevent accidental operations that would destabilize the cluster. More information can be found in the documentation for that setting.

The first time a cluster is started, cluster.initial_master_nodes must be set to perform cluster bootstrapping. It should contain the names of the master-eligible nodes in the initial cluster and be defined on every master-eligible node in the cluster. See the discovery settings summary for an example, and the cluster bootstrapping reference documentation describes this setting in more detail.

The discovery.zen.minimum_master_nodes setting is permitted, but ignored, on 7.x nodes.

If you wish to remove half or more of the master-eligible nodes from a cluster, you must first exclude the affected nodes from the voting configuration using the voting config exclusions API. If you remove fewer than half of the master-eligible nodes at the same time, voting exclusions are not required. If you remove only master-ineligible nodes such as data-only nodes or coordinating-only nodes, voting exclusions are not required. Likewise, if you add nodes to the cluster, voting exclusions are not required.

Production deployments of Elasticsearch now require at least one of the following settings to be specified in the elasticsearch.yml configuration file:

The first three settings in this list are only available in versions 7.0 and above. If you are preparing to upgrade from an earlier version, you must set discovery.zen.ping.unicast.hosts or discovery.zen.hosts_provider.

The discovery.zen.no_master_block setting is now known as cluster.no_master_block. Any value set for discovery.zen.no_master_block is now ignored. You should remove this setting and, if needed, set cluster.no_master_block appropriately after the upgrade.

By default the cluster fault detection subsystem now considers a node to be faulty if it fails to respond to 3 consecutive pings, each of which times out after 10 seconds. Thus a node that is unresponsive for longer than 30 seconds is liable to be removed from the cluster. Previously the default timeout for each ping was 30 seconds, so that an unresponsive node might be kept in the cluster for over 90 seconds.

In earlier versions it was possible to use master-ineligible nodes during the discovery process, either as seed nodes or to transfer discovery gossip indirectly between the master-eligible nodes. Clusters that relied on master-ineligible nodes like this were fragile and unable to automatically recover from some kinds of failure. Discovery now involves only the master-eligible nodes in the cluster so that it is not possible to rely on master-ineligible nodes like this. You should configure discovery.seed_hosts or another seed hosts provider to provide the addresses of all the master-eligible nodes in your cluster.

Previously, some ingest configuration exception information about ingest processors was sent to the client in the HTTP headers, which is inconsistent with how exceptions are conveyed in other parts of Elasticsearch.

Configuration exception information is now conveyed as a field in the response body.

There was some special handling for installing and removing the ingest-geoip and ingest-user-agent plugins after they were converted to modules. This special handling was done to minimize breaking users in a minor release, and would exit with a status code zero to avoid breaking automation.

This special handling has now been removed.

Previous versions of Elasticsearch defaulted to creating five shards per index. Starting with 7.0.0, the default is now one shard per index.

Due to cross-cluster search using : to separate a cluster and index name, index names may no longer contain :.

Negative values were interpreted as zero in earlier versions but are no longer accepted.

In previous versions issuing a _flush or _force_merge (with flush=true) had the undocumented side-effect of refreshing the index which made new documents visible to searches and non-realtime GET operations. From now on these operations don’t have this side-effect anymore. To make documents visible an explicit _refresh call is needed unless the index is refreshed by the internal scheduler.

To safeguard against creating too many index terms, the difference between max_ngram and min_ngram in NGramTokenFilter and NGramTokenizer has been limited to 1. This default limit can be changed with the index setting index.max_ngram_diff. Note that if the limit is exceeded a error is thrown only for new indices. For existing pre-7.0 indices, a deprecation warning is logged.

To safeguard against creating too many tokens, the difference between max_shingle_size and min_shingle_size in ShingleTokenFilter has been limited to 3. This default limit can be changed with the index setting index.max_shingle_diff. Note that if the limit is exceeded a error is thrown only for new indices. For existing pre-7.0 indices, a deprecation warning is logged.

Indices created with version 7.0.0 onwards will have an automatic index.number_of_routing_shards value set. This might change how documents are distributed across shards depending on how many shards the index has. In order to maintain the exact same distribution as a pre 7.0.0 index, the index.number_of_routing_shards must be set to the index.number_of_shards at index creation time. Note: if the number of routing shards equals the number of shards _split operations are not supported.

Shards belonging to an index that does not have an explicit index.refresh_interval configured will no longer refresh in the background once the shard becomes "search idle", ie the shard hasn’t seen any search traffic for index.search.idle.after seconds (defaults to 30s). Searches that access a search idle shard will be "parked" until the next refresh happens. Indexing requests with wait_for_refresh will also trigger a background refresh.

The following previously deprecated url parameter have been removed:

Previously the in flight requests circuit breaker considered only the raw byte representation. By bumping the value of network.breaker.inflight_requests.overhead from 1 to 2, this circuit breaker considers now also the memory overhead of representing the request as a structured object.

The parent circuit breaker defines a new setting indices.breaker.total.use_real_memory which is true by default. This means that the parent circuit breaker will trip based on currently used heap memory instead of only considering the reserved memory by child circuit breakers. When this setting is true, the default parent breaker limit also changes from 70% to 95% of the JVM heap size. The previous behavior can be restored by setting indices.breaker.total.use_real_memory to false.

As doc values have been enabled by default in earlier versions of Elasticsearch, there is less need for fielddata. Therefore, the default value of the setting indices.breaker.fielddata.limit has been lowered from 60% to 40% of the JVM heap size.

Deprecated option value fix for setting index.shard.check_on_startup is not supported.

Use the elasticsearch-shard tool to remove corrupted translog data.

The _all field deprecated in 6 have now been removed.

This field used to index a composite key formed of the _type and the _id. Now that indices cannot have multiple types, this has been removed in favour of _id.

The _default_ mapping has been deprecated in 6.0 and is now no longer allowed in 7.0. Trying to configure a _default_ mapping on 7.x indices will result in an error.

If an index template contains a _default_ mapping, it will fail to create new indices. To resolve this issue, the _default_ mapping should be removed from the template. Note that in 7.x, the get template API does not show the _default_ mapping by default, even when it is defined in the mapping. To see all mappings in the template, the include_type_name parameter must be supplied:

GET /_template/my_template?include_type_name

For more details on the include_type_name parameter and other types-related API changes, please see Removal of mapping types.

The index_options field for numeric fields has been deprecated in 6 and has now been removed.

To safeguard against out of memory errors, the number of nested json objects within a single document across all fields has been limited to 10000. This default limit can be changed with the index setting index.mapping.nested_objects.limit.

This option is useless now that all indices have at most one type.

The classic similarity relied on coordination factors for scoring to be good in presence of stopwords in the query. This feature has been removed from Lucene, which means that the classic similarity now produces scores of lower quality. It is advised to switch to BM25 instead, which is widely accepted as a better alternative.

An error will now be thrown when unknown configuration options are provided to similarities. Such unknown parameters were ignored before.

geo_shape types now default to using a vector indexing approach based on Lucene’s new LatLonShape field type. This indexes shapes as a triangular mesh instead of decomposing them into individual grid cells. To index using legacy prefix trees the tree parameter must be explicitly set to one of quadtree or geohash. Note that these strategies are now deprecated and will be removed in a future version.

IMPORTANT NOTE: If using timed index creation from templates, the geo_shape mapping should also be changed in the template to explicitly define tree to one of geohash or quadtree. This will ensure compatibility with previously created indexes.

The following type parameters are deprecated for the geo_shape field type: tree, precision, tree_levels, distance_error_pct, points_only, and strategy. They will be removed in a future version.

The maximum allowed number of completion contexts in a mapping will be limited to 10 in the next major version. Completion fields that define more than 10 contexts in a mapping will log a deprecation warning in this version.

The default for include_type_name is now false for all APIs that accept the parameter.

Types have been removed from the datafeed config and are no longer valid parameters.

The terms dictionary is the part of the inverted index that records all terms that occur within a segment in sorted order. In order to provide fast retrieval, terms dictionaries come with a small terms index that allows for efficient random access by term. Until now this terms index had always been loaded on-heap.

As of 7.0, the terms index is loaded on-heap for fields that only have unique values such as _id fields, and off-heap otherwise - likely most other fields. This is expected to reduce memory requirements but might slow down search requests if both below conditions are met:

This change affects both existing indices created with Elasticsearch 6.x and new indices created with Elasticsearch 7.x.

Adaptive replica selection has been enabled by default. If you wish to return to the older round robin of search requests, you can use the cluster.routing.use_adaptive_replica_selection setting:

PUT /_cluster/settings
{
    "transient": {
        "cluster.routing.use_adaptive_replica_selection": false
    }
}

The Search API returns 400 - Bad request while it would previously return 500 - Internal Server Error in the following cases of invalid request:

Setting request_cache:true on a query that creates a scroll (scroll=1m) has been deprecated in 6 and will now return a 400 - Bad request. Scroll queries are not meant to be cached.

Including a rescore clause on a query that creates a scroll (scroll=1m) has been deprecated in 6.5 and will now return a 400 - Bad request. Allowing rescore on scroll queries would break the scroll sort. In the 6.x line, the rescore clause was silently ignored (for scroll queries), and it was allowed in the 5.x line.

The following string distance algorithms were given additional names in 6.2 and their existing names were deprecated. The deprecated names have now been removed.

The popular mode for Suggesters (term and phrase) now uses the doc frequency (instead of the sum of the doc frequency) of the input terms to compute the frequency threshold for candidate suggestions.

Executing a Terms Query with a lot of terms may degrade the cluster performance, as each additional term demands extra processing and memory. To safeguard against this, the maximum number of terms that can be used in a Terms Query request has been limited to 65536. This default maximum can be changed for a particular index with the index setting index.max_terms_count.

Executing a Regexp Query with a long regex string may degrade search performance. To safeguard against this, the maximum length of regex that can be used in a Regexp Query request has been limited to 1000. This default maximum can be changed for a particular index with the index setting index.max_regex_length.

Executing queries that use automatic expansion of fields (e.g. query_string, simple_query_string or multi_match) can have performance issues for indices with a large numbers of fields. To safeguard against this, a default limit of 1024 fields has been introduced for queries using the "all fields" mode ("default_field": "*") or other fieldname expansions (e.g. "foo*"). If needed, you can change this limit using the indices.query.bool.max_clause_count static cluster setting.

Search requests with extra content after the main object will no longer be accepted by the _search endpoint. A parsing exception will be thrown instead.

The format of doc-value fields is changing to be the same as what could be obtained in 6.x with the special use_field_mapping format. This is mostly a change for date fields, which are now formatted based on the format that is configured in the mappings by default. This behavior can be changed by specifying a format within the doc-value field.

The ability to query and index context enabled suggestions without context, deprecated in 6.x, has been removed. Context enabled suggestion queries without contexts have to visit every suggestion, which degrades the search performance considerably.

For geo context the value of the path parameter is now validated against the mapping, and the context is only accepted if path points to a field with geo_point type.

max_concurrent_shard_requests used to limit the total number of concurrent shard requests a single high level search request can execute. In 7.0 this changed to be the max number of concurrent shard requests per node. The default is now 5.

max_score used to be set to 0 whenever scores are not tracked. null is now used instead which is a more appropriate value for a scenario where scores are not available.

Setting a negative boost for a query or a field, deprecated in 6x, is not allowed in this version. To deboost a specific query or field you can use a boost comprise between 0 and 1.

Negative scores in the Function Score Query are deprecated in 6.x, and are not allowed in this version. If a negative score is produced as a result of computation (e.g. in script_score or field_value_factor functions), an error will be thrown.

The filter context has been removed from Elasticsearch’s query builders, the distinction between queries and filters is now decided in Lucene depending on whether queries need to access score or not. As a result bool queries with should clauses that don’t need to access the score will no longer set their minimum_should_match to 1. This behavior has been deprecated in the previous major version.

The total hits that match the search request is now returned as an object with a value and a relation. value indicates the number of hits that match and relation indicates whether the value is accurate (eq) or a lower bound (gte):

{
    "hits": {
        "total": {
            "value": 1000,
            "relation": "eq"
        },
        ...
    }
}

The total object in the response indicates that the query matches exactly 1000 documents ("eq"). The value is always accurate ("relation": "eq") when track_total_hits is set to true in the request. You can also retrieve hits.total as a number in the rest response by adding rest_total_hits_as_int=true in the request parameter of the search request. This parameter has been added to ease the transition to the new format and will be removed in the next major version (8.0).

If track_total_hits is set to false in the search request the search response will set hits.total to null and the object will not be displayed in the rest layer. You can add rest_total_hits_as_int=true in the search request parameters to get the old format back ("total": -1).

By default search request will count the total hits accurately up to 10,000 documents. If the total number of hits that match the query is greater than this value, the response will indicate that the returned value is a lower bound:

{
     "_shards": ...
     "timed_out": false,
     "took": 100,
     "hits": {
         "max_score": 1.0,
         "total" : {
             "value": 10000,    
             "relation": "gte"  
         },
         "hits": ...
     }
}

You can force the count to always be accurate by setting track_total_hits to true explicitly in the search request.

Lucene 8 introduced more constraints on similarities, in particular:

Negative weight parameters in the function_score are no longer allowed.

The number of automatically expanded fields for the "all fields" mode ("default_field": "*") for the query_string and simple_query_string queries is now 1024 fields.

Plugins must now explicitly indicate the type of suggestion that they produce.

Previously, the laplace smoothing used by the phrase suggester added alpha, when it should instead multiply. This behavior has been changed and will affect suggester scores.

The systemd service file /usr/lib/systemd/system/elasticsearch.service was previously marked as a configuration file in rpm and deb packages. Overrides to the systemd elasticsearch service should be made in /etc/systemd/system/elasticsearch.service.d/override.conf.

The tar package previously included files in the bin directory meant only for windows. These files have been removed. Use the zip package instead.

Ubuntu 14.04 will reach end-of-life on April 30, 2019. As such, we are no longer supporting Ubuntu 14.04.

The ability to use ${prompt.secret} and ${prompt.text} to collect secrets from the CLI at server start is no longer supported. Secure settings have replaced the need for these prompts.

See Azure Repository settings.

See Google Cloud Storage Client Settings.

As a consequence of the change to Realm settings, the getRealmSettings method has been removed from the SecurityExtension class, and the settings method on RealmConfig now returns the node’s (global) settings. Custom security extensions should register their settings by implementing the standard Plugin.getSettings method, and can retrieve them from RealmConfig.settings() or using one of the RealmConfig.getSetting methods. Each realm setting should be defined as an AffixSetting as shown in the example below:

Setting.AffixSetting<String> MY_SETTING = Setting.affixKeySetting(
  "xpack.security.authc.realms." + MY_REALM_TYPE + ".", "my_setting",
  key -> Setting.simpleString(key, properties)
);

The RealmSettings.simpleString method can be used as a convenience for the above.

Tribe node functionality has been removed in favor of Cross Cluster Search.

Hipchat has been deprecated and shut down as a service. The hipchat action for watches has been removed.

Elasticsearch maintains a numeric version field for each document it stores. That field is incremented by one with every change to the document. Until 7.0.0 the API allowed using that field for optimistic concurrency control, i.e., making a write operation conditional on the current document version. Sadly, that approach is flawed because the value of the version doesn’t always uniquely represent a change to the document. If a primary fails while handling a write operation, it may expose a version that will then be reused by the new primary.

Due to that issue, internal versioning can no longer be used and is replaced by a new method based on sequence numbers. See Optimistic concurrency control for more details.

Note that the external versioning type is still fully supported.

A number of duplicate parameters deprecated in 6.x have been removed from Bulk request, Multi Get request, Term Vectors request, and More Like This Query requests.

The following camel case parameters have been removed:

The following parameters starting with underscore have been removed:

Instead of these removed parameters, use their non camel case equivalents without starting underscore, e.g. use version_type instead of _version_type or versionType.

In previous versions of Elasticsearch, the thread pool info returned in the nodes info API returned min and max values reflecting the configured minimum and maximum number of threads that could be in each thread pool. The trouble with this representation is that it does not align with the configuration parameters used to configure thread pools. For scaling thread pools, the minimum number of threads is configured by a parameter called core and the maximum number of threads is configured by a parameter called max. For fixed thread pools, there is only one configuration parameter along these lines and that parameter is called size, reflecting the fixed number of threads in the pool. This discrepancy between the API and the configuration parameters has been rectified. Now, the API will report core and max for scaling thread pools, and size for fixed thread pools.

Similarly, in the cat thread pool API the existing size output has been renamed to pool_size which reflects the number of threads currently in the pool; the shortcut for this value has been changed from s to psz. The min output has been renamed to core with a shortcut of cr, the shortcut for max has been changed to mx, and the size output with a shortcut of sz has been reused to report the configured number of threads in the pool. This aligns the output of the API with the configuration values for thread pools. Note that core and max will be populated for scaling thread pools, and size will be populated for fixed thread pools.

and Update request. The Update API returns 400 - Bad request if request contains unknown parameters (instead of ignored in the previous version).

If you attempt to PUT a document with versioning (e.g. PUT /test/_doc/1?version=4) but the document does not exist, a cryptic message is returned:

version conflict, current version [-1] is different than the one provided [4]

Now if the document is missing a more helpful message is returned:

document does not exist (expected version [4])

Although exceptions messages are liable to change and not generally subject to backwards compatibility, the nature of this message might mean clients are relying on parsing the version numbers and so the format change might impact some users.

Previously, suggest stats were folded into search stats. Support for the suggest metric on the indices stats and nodes stats APIs remained for backwards compatibility. Backwards support for the suggest metric was deprecated in 6.3.0 and now removed in 7.0.0.

In the past, fields could be provided either as a parameter, or as part of the request body. Specifying fields in the request body as opposed to a parameter was deprecated in 6.4.0, and is now unsupported in 7.0.0.

Versions of Elasticsearch prior to 6.4.0 did not copy index settings on shrink and split operations. Starting with Elasticsearch 7.0.0, the default behavior will be for such settings to be copied on such operations. To enable users in 6.4.0 to transition in 6.4.0 to the default behavior in 7.0.0, the copy_settings parameter was added on the REST layer. As this behavior will be the only behavior in 8.0.0, this parameter is deprecated in 7.0.0 for removal in 8.0.0.

When putting stored scripts, support for storing them with the deprecated template context or without a context is now removed. Scripts must be stored using the script context as mentioned in the documentation.

The behavior and response codes of the get aliases API no longer vary depending on whether security features are enabled. Previously a 404 - NOT FOUND (IndexNotFoundException) could be returned in case the current user was not authorized for any alias. An empty response with status 200 - OK is now returned instead at all times.

The Put User API response was changed in 6.5.0 to add the created field outside of the user object where it previously had been. In 7.0.0 the user object has been removed in favor of the top level created field.

The deprecated in 6.x url parameters are now removed. Use _source_includes and _source_excludes instead.

MultiSearchRequests issued through _msearch now validate all keys in the metadata section. Previously unknown keys were ignored while now an exception is thrown.

The deprecated graph endpoints (those with /_graph/_explore) have been removed.

The _termvector endpoint was deprecated in 2.0 and has now been removed. The endpoint _termvectors (plural) should be used instead.

Restricted indices (currently only .security-6 and .security) are special internal indices that require setting the allow_restricted_indices flag on every index permission that covers them. If this flag is false (default) the permission will not cover these and actions against them will not be authorized. However, the monitoring APIs were the only exception to this rule. This exception has been forfeited and index monitoring privileges have to be granted explicitly, using the allow_restricted_indices flag on the permission (as any other index privilege).

The _cache/clear API no longer supports the GET HTTP verb. It must be called with POST.

The compressed_size / compressed_size_in_bytes fields were removed from the Cluster State API response. The calculation of the size was expensive and had dubious value, so the field was removed from the response.

The Migration Assistance API has been functionally replaced by the Deprecation Info API, and the Migration Upgrade API is not used for the transition from ES 6.x to 7.x, and does not need to be kept around to repair indices that were not properly upgraded before upgrading the cluster, as was the case in 6.

The thread_pool information returned from the Nodes and Cat APIs has been standardized to use the same terminology as the thread pool configurations. This means the response will align with the configuration instead of being the same across all the thread pools, regardless of type.

If the cluster was configured with no_master_block: write and lost its master, it would return a 503 status code from a main request (GET /) even though there are viable read-only nodes available. The cluster now returns 200 status in this situation.

Clearing the cache indices could previously be done via GET and POST. As GET should only support read only non state-changing operations, this is no longer allowed. Only POST can be used to clear the cache.

isShardsAcked has been replaced by isShardsAcknowledged in CreateIndexResponse, RolloverResponse and CreateIndexClusterStateUpdateResponse.

The prepareExecute method which created a request builder has been removed from the client api. Instead, construct a builder for the appropriate request directly.

The variants of Retry.withBackoff that included Settings have been removed because Settings is no longer needed.

The client method termVector, deprecated in 2.0, has been removed. The method termVectors (plural) should be used instead.

The constructor AbstractLifecycleComponent(Settings settings), deprecated in 6.7 has been removed. The parameterless constructor should be used instead.

Geometry classes used to represent geo values in SQL have been moved from the org.elasticsearch.geo.geometry package to the org.elasticsearch.geometry package and the order of the constructor parameters has changed from lat, lon to lon, lat.

node.name now defaults to the hostname at the time when Elasticsearch is started. Previously the default node name was the first eight characters of the node id. It can still be configured explicitly in elasticsearch.yml.

The cross-cluster search remote cluster connection infrastructure is also used in cross-cluster replication. This means that the setting names search.remote.* used for configuring cross-cluster search belie the fact that they also apply to other situations where a connection to a remote cluster as used. Therefore, these settings have been renamed from search.remote.* to cluster.remote.*. For backwards compatibility purposes, we will fallback to search.remote.* if cluster.remote.* is not set. For any such settings stored in the cluster state, or set on dynamic settings updates, we will automatically upgrade the setting from search.remote.* to cluster.remote.*. The fallback settings will be removed in 8.0.0.

The following settings have been removed:

The new settings have the same meaning as the removed ones, but the prefix name component is no longer meaningful as logfile audit entries are structured JSON documents and are not prefixed by anything. Moreover, xpack.security.audit.logfile.emit_node_name has changed its default from true to false. All other settings mentioned before, have kept their default value of false.

The settings for all security realms must now include the realm type as part of the setting name, and the explicit type setting has been removed.

A realm that was previous configured as:

xpack.security.authc.realms:
  ldap1:
    type: ldap
    order: 1
    url: "ldaps://ldap.example.com/"

Must be migrated to:

xpack.security.authc.realms:
  ldap.ldap1:
    order: 1
    url: "ldaps://ldap.example.com/"

Any realm specific secure settings that have been stored in the elasticsearch keystore (such as ldap bind passwords, or passwords for ssl keys) must be updated in a similar way.

The default TLS/SSL settings, which were prefixed by xpack.ssl, have been removed. The removal of these default settings also removes the ability for a component to fallback to a default configuration when using TLS. Each component (realm, transport, http, http client, etc) must now be configured with their own settings for TLS if it is being used.

TLS version 1.0 is now disabled by default as it suffers from known security issues. The default protocols are now TLSv1.3 (if supported), TLSv1.2 and TLSv1.1.

You can enable TLS v1.0 by configuring the relevant ssl.supported_protocols setting to include "TLSv1". Depending on your local configuration and the TLS protocols that are in use on your network, you may need to enable TLS v1.0 support in any or all of the following places:

The following is an example that enables TLS v1.0 for incoming HTTP connections:

xpack.security.http.ssl.supported_protocols: [ "TLSv1.3", "TLSv1.2", "TLSv1.1", "TLSv1" ]

On trial licenses, xpack.security.enabled defaults to false.

In prior versions, a trial license would automatically enable security if either

This behaviour has been now removed, so security is only enabled if:

The following settings have been removed in favor of the secure variants. The secure settings have to be defined inside each cluster node’s keystore, i.e., they are not to be specified via the cluster settings API.

All the settings under the xpack.security.audit.index namespace have been removed. In addition, the xpack.security.audit.outputs setting has been removed as well.

These settings enabled and configured the audit index output type. This output type has been removed because it was unreliable in certain scenarios and this could have lead to dropping audit events while the operations on the system were allowed to continue as usual. The recommended replacement is the use of the logfile audit output type and using other components from the Elastic Stack to handle the indexing part.

ECS format is now the default. The ecs setting for the user agent ingest processor now defaults to true.

The action.master.force_local setting was an undocumented setting, used internally by the tribe node to force reads to local cluster state (instead of forwarding to a master, which tribe nodes did not have). Since the tribe node was removed, this setting was removed too.

The cluster-wide shard limit is now enforced and not optional. The limit can still be adjusted as desired using the cluster settings API.

Previously, http.max_content_length would reset to 100mb if the setting was greater than Integer.MAX_VALUE. This leniency has been removed.

Fields of type long and date had getDate() and getDates() methods (for multi valued fields) to get an object with date specific helper methods for the current doc value. In 5.3.0, date fields were changed to expose this same date object directly when calling doc["myfield"].value, and the getter methods for date objects were deprecated. These methods have now been removed. Instead, use .value on date fields, or explicitly parse long fields into a date object using Instance.ofEpochMillis(doc["myfield"].value).

doc['field'].value will throw an exception if the document is missing a value for the field field.

To check if a document is missing a value, you can use doc['field'].size() == 0.

Malformed scripts, either in search templates, ingest pipelines or search requests, return 400 - Bad request while they would previously return 500 - Internal Server Error. This also applies for stored scripts.

The ScriptDocValues#getValues() method is deprecated in 6.6 and will be removed in 7.0. Use doc["foo"] in place of doc["foo"].values.

Snapshot stats details are provided in a new structured way:

All API methods accepting headers as a Header varargs argument, deprecated since 6.4, have been removed in favour of the newly introduced methods that accept instead a RequestOptions argument. In case you are not specifying any header, e.g. client.index(indexRequest) becomes client.index(indexRequest, RequestOptions.DEFAULT). In case you are specifying headers e.g. client.index(indexRequest, new Header("name" "value")) becomes client.index(indexRequest, RequestOptions.DEFAULT.toBuilder().addHeader("name", "value").build());

The Cluster Health API used to default to shards level to ease migration from transport client that doesn’t support the level parameter and always returns information including indices and shards details. The level default value has been aligned with the Elasticsearch default level: cluster.

RestClient and RestClientBuilder no longer support the maxRetryTimeout setting. The setting was removed as its counting mechanism was not accurate and caused issues while adding little value.

We deprecated the flavors of performRequest and performRequestAsync that do not take Request objects in 6.4.0 in favor of the flavors that take Request objects because those methods can be extended without breaking backwards compatibility.

We deprecated setHosts in 6.4.0 in favor of setNodes because it supports host metadata used by the NodeSelector.

The minimum compiler version on the low-level REST client has been bumped to JDK 8.

Elasticsearch now will produce additional log files in JSON format. They will be stored in *.json suffix files. Following files should be expected now in log directory: * ${cluster_name}_server.json * ${cluster_name}_deprecation.json * ${cluster_name}_index_search_slowlog.json * ${cluster_name}_index_indexing_slowlog.json * ${cluster_name}.log * ${cluster_name}_deprecation.log * ${cluster_name}_index_search_slowlog.log * ${cluster_name}_index_indexing_slowlog.log * ${cluster_name}_audit.json * gc.log

Note: You can configure which of these files are written by editing log4j2.properties.

Log files with the .log file extension using the old pattern layout format are now considered deprecated and the newly added JSON log file format with the .json file extension should be used instead. Note: GC logs which are written to the file gc.log will not be changed.

All Docker console logs are now in JSON format. You can distinguish logs streams with the type field.

Elasticsearch no longer produces the ${cluster_name}_access.log plaintext audit log file. The ${cluster_name}_audit.log files also no longer exist; they are replaced by ${cluster_name}_audit.json files. When auditing is enabled, auditing events are stored in these dedicated JSON log files on each node.

Repurposing an existing node by changing node.master or node.data to false can leave lingering on-disk metadata and data around, which will not be accessible by the node’s new role. Beside storing non-accessible data, this can lead to situations where dangling indices are imported even though the node might not be able to host any shards, leading to a red cluster health. To avoid this,

Beware that such role changes done prior to the 7.0 upgrade could prevent node start up in 7.0.

Since Java 8 there is a dedicated java.time package, which is superior to the Joda-Time library, that has been used so far in Elasticsearch. One of the biggest advantages is the ability to be able to store dates in a higher resolution than milliseconds for greater precision. Also this will allow us to remove the Joda-Time dependency in the future.

The mappings, aggregations and search code switched from Joda-Time to java time.

With the release of Elasticsearch 6.7 a backwards compatibility layer was introduced, that checked if you are using a Joda-Time based formatter, that is supported differently in java time. A log message was emitted, and you could create the proper java time based formatter prefixed with an 8.

With Elasticsearch 7.0 all formatters are now java based, which means you will get exceptions when using deprecated formatters without checking the deprecation log in 6.7. In the worst case you may even end up with different dates.

An example deprecation message looks like this, that is returned, when you try to use a date formatter that includes a lower case Y

Use of 'Y' (year-of-era) will change to 'y' in the next major version of
Elasticsearch. Prefix your date format with '8' to use the new specifier.

So, instead of using YYYY.MM.dd you should use 8yyyy.MM.dd.

You can find more information about available formatting strings in the DateTimeFormatter javadocs.

The epoch_millis and epoch_second formatters no longer support scientific notation.

If you are using the century of era formatter in a date (C), this will no longer be supported.

The year-of-era formatting character is a Y in Joda-Time, but a lowercase y in java time.

The week-based-year formatting character is a lowercase x in Joda-Time, but an upper-case Y in java time.

Timezones have to be specified as java time based zone objects. This means, instead of using a org.joda.time.DateTimeZone the use of java.time.ZoneId is required.

Examples of possible uses are the QueryStringQueryBuilder, the RangeQueryBuilder or the DateHistogramAggregationBuilder, each of them allow for an optional timezone for that part of the search request.

The date based aggregation buckets in responses used to be of type JodaTime. Due to migrating to java-time, the buckets are now of type ZonedDateTime. As the client is returning untyped objects here, you may run into class cast exceptions only when running the code, but not at compile time, ensure you have proper test coverage for this in your own code.

When you are running Elasticsearch 7 with Java 8, you are not able to parse the timezone GMT0 properly anymore. The reason for this is a bug in the JDK, which has not been fixed for JDK8. You can read more in the official issue This bug is fixed in JDK9 and later versions.

If dates are used in scripting, a backwards compatibility layer has been added that emulates the Joda-Time methods, but logs a deprecation message as well to use the java time methods.

The following methods will be removed in future versions of Elasticsearch and should be replaced.

With the switch to java time, support for negative timestamps has been removed. For dates before 1970, use a date format containing a year.

For a detailed migration guide, see Java time migration guide.