Elasticsearch breaking changes

edit

This list summarizes the most important breaking changes in Elasticsearch 7.0.1. For the complete list, go to Elasticsearch breaking changes.

Indices created before 7.0

edit

Elasticsearch 7.0 can read indices created in version 6.0 or above. An Elasticsearch 7.0 node will not start in the presence of indices created in a version of Elasticsearch before 6.0.

Reindex indices from Elasticsearch 5.x or before

Indices created in Elasticsearch 5.x or before will need to be reindexed with Elasticsearch 6.x in order to be readable by Elasticsearch 7.x.

search.max_buckets in the cluster setting

edit

The dynamic cluster setting named search.max_buckets now defaults to 10,000 (instead of unlimited in the previous version). Requests that try to return more than the limit will fail with an exception.

Internal Versioning is no longer supported for optimistic concurrency control

edit

Elasticsearch maintains a numeric version field for each document it stores. That field is incremented by one with every change to the document. Until 7.0.0 the API allowed using that field for optimistic concurrency control, i.e., making a write operation conditional on the current document version. Sadly, that approach is flawed because the value of the version doesn’t always uniquely represent a change to the document. If a primary fails while handling a write operation, it may expose a version that will then be reused by the new primary.

Due to that issue, internal versioning can no longer be used and is replaced by a new method based on sequence numbers. See Optimistic concurrency control for more details.

Note that the external versioning type is still fully supported.

Shard preferences _primary, _primary_first, _replica, and _replica_first are removed

edit

These shard preferences are removed in favour of the _prefer_nodes and _only_nodes preferences.

Cluster-wide shard soft limit

edit

Clusters now have soft limits on the total number of open shards in the cluster based on the number of nodes and the cluster.max_shards_per_node cluster setting, to prevent accidental operations that would destabilize the cluster. More information can be found in the documentation for that setting.

Cluster bootstrapping is required if discovery is configured

edit

The first time a cluster is started, cluster.initial_master_nodes must be set to perform cluster bootstrapping. It should contain the names of the master-eligible nodes in the initial cluster and be defined on every master-eligible node in the cluster. See the discovery settings summary for an example, and the cluster bootstrapping reference documentation describes this setting in more detail.

The discovery.zen.minimum_master_nodes setting is permitted, but ignored, on 7.x nodes.

Discovery configuration is required in production

edit

Production deployments of Elasticsearch now require at least one of the following settings to be specified in the elasticsearch.yml configuration file:

  • discovery.seed_hosts
  • discovery.seed_providers
  • cluster.initial_master_nodes
  • discovery.zen.ping.unicast.hosts
  • discovery.zen.hosts_provider

The first three settings in this list are only available in versions 7.0 and above. If you are preparing to upgrade from an earlier version, you must set discovery.zen.ping.unicast.hosts or discovery.zen.hosts_provider.

Index creation no longer defaults to five shards

edit

Previous versions of Elasticsearch defaulted to creating five shards per index. Starting with 7.0.0, the default is now one shard per index.

Ingest configuration exception information is now transmitted in metadata field

edit

Previously, some ingest configuration exception information about ingest processors was sent to the client in the HTTP headers, which is inconsistent with how exceptions are conveyed in other parts of Elasticsearch.

Configuration exception information is now conveyed as a field in the response body.

Replacing Joda-Time with java time

edit

Since Java 8 there is a dedicated java.time package, which is superior to the Joda-Time library, that has been used so far in Elasticsearch. One of the biggest advantages is the ability to be able to store dates in a higher resolution than milliseconds for greater precision. Also this will allow us to remove the Joda-Time dependency in the future.

The mappings, aggregations and search code switched from Joda-Time to java time.

The _default_ mapping is no longer allowed

edit

The _default_ mapping has been deprecated in 6.0 and is now no longer allowed in 7.0. Trying to configure a _default_ mapping on 7.x indices will result in an error.

Tribe node removed

edit

Tribe node functionality has been removed in favor of cross-cluster search.

hits.total is now an object in the search response

edit

The total hits that match the search request is now returned as an object with a value and a relation. value indicates the number of hits that match and relation indicates whether the value is accurate (eq) or a lower bound (gte):

{
    "hits": {
        "total": {
            "value": 1000,
            "relation": "eq"
        },
        ...
    }
}

The total object in the response indicates that the query matches exactly 1000 documents ("eq"). The value is always accurate ("relation": "eq") when track_total_hits is set to true in the request. You can also retrieve hits.total as a number in the rest response by adding rest_total_hits_as_int=true in the request parameter of the search request. This parameter has been added to ease the transition to the new format and will be removed in the next major version (8.0).

track_total_hits defaults to 10,000

edit

By default search request will count the total hits accurately up to 10,000 documents. If the total number of hits that match the query is greater than this value, the response will indicate that the returned value is a lower bound:

{
     "_shards": ...
     "timed_out": false,
     "took": 100,
     "hits": {
         "max_score": 1.0,
         "total" : {
             "value": 10000,    
             "relation": "gte"  
         },
         "hits": ...
     }
}

There are at least 10000 documents that match the query

This is a lower bound ("gte").

You can force the count to always be accurate by setting "track_total_hits to true explicitly in the search request.

The default for node.name is now the hostname

edit

node.name now defaults to the hostname at the time when Elasticsearch is started. Previously the default node name was the first eight characters of the node id. It can still be configured explicitly in elasticsearch.yml.

Security realms settings

edit

The settings for all security realms must now include the realm type as part of the setting name, and the explicit type setting has been removed.

A realm that was previous configured as:

xpack.security.authc.realms:
  ldap1:
    type: ldap
    order: 1
    url: "ldaps://ldap.example.com/"

Must be migrated to:

xpack.security.authc.realms:
  ldap.ldap1:
    order: 1
    url: "ldaps://ldap.example.com/"

Any realm specific secure settings that have been stored in the elasticsearch keystore (such as ldap bind passwords, or passwords for ssl keys) must be updated in a similar way.

TLS v1.0 disabled

edit

TLS version 1.0 is now disabled by default as it suffers from known security issues. The default protocols are now TLSv1.3 (if supported), TLSv1.2 and TLSv1.1.

You can enable TLS v1.0 by configuring the relevant ssl.supported_protocols setting to include "TLSv1". Depending on your local configuration and the TLS protocols that are in use on your network, you may need to enable TLS v1.0 support in any or all of the following places:

xpack.security.http.ssl.supported_protocols
For incoming HTTP connections to Elasticsearch’s HTTP (Rest) interface. If there are clients that connect to Elasticsearch and do not support newer TLS versions, you must update this setting.
xpack.http.ssl.supported_protocols
For outgoing HTTP connections from Watcher. If you have watches that connect to external HTTP servers and do not support newer TLS versions, you must update this setting.
xpack.security.authc.realms.ldap.{name}.ssl.supported_protocols
For outgoing LDAP connections from Elasticsearch security features. If you have an LDAP realm enabled and the LDAP directory to which that realm connects does not support newer TLS versions, you must update this setting.
xpack.security.authc.realms.active_directory.{name}.ssl.supported_protocols
For outgoing Active Directory (LDAP) connections from Elasticsearch security features. If you have an AD realm enabled and the directory server to which that realm connects does not support newer TLS versions, you must update this setting.
xpack.security.authc.realms.saml.{name}.ssl.supported_protocols
For outgoing HTTP connections to retrieve SAML metadata. If you have a SAML realm enabled and the realm is configured to retrieve metadata over HTTPS (that is, idp.metadata.path is a URL starting with https://) and the web server which hosts the metadata does not support newer TLS versions, you must update this setting.
xpack.security.authc.realms.oidc.{name}.ssl.supported_protocols
For outgoing HTTP connections to an OpenId Connect Provider. If you have an OpenId Connect ("oidc") realm enabled and the realm is configured to connect to a remote OpenID Connect Provider which does not support newer TLS versions, you must update this setting.
xpack.monitoring.exporters.{name}.ssl.supported_protocols
For remote monitoring data. If your monitoring data is exported to a remote monitoring cluster and that cluster is configured to only support TLSv1, you must update this setting.
reindex.ssl.supported_protocols
For reindex from remote. If you reindex data from a remote Elasticsearch cluster which has SSL enabled on the http interface and that cluster is configured to only support TLSv1, you must update this setting.
xpack.security.transport.ssl.supported_protocols
For incoming connections between Elasticsearch nodes. If you have specialized network equipment which inspects TLS packets between your nodes, and that equipment enforces TLSv1 you must update this setting.

The following is an example that enables TLS v1.0 for incoming HTTP connections:

xpack.security.http.ssl.supported_protocols: [ "TLSv1.3", "TLSv1.2", "TLSv1.1", "TLSv1" ]