Upgrade to Elasticsearch 5.x

Version 5.6 of Elasticsearch has passed its EOL date. This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade your deployment to a more current version of the Elastic stack.

Elasticsearch 5.x provides major new features and improved usability, but there are a few things you need to keep in mind when upgrading to our latest and greatest software in Elasticsearch Add-On for Heroku.

Breaking changes in Elasticsearch 5.x

edit

A number of Elasticsearch queries were deprecated in version 2.0 or later and removed in version 5.x. Applications running deprecated queries might break after upgrading to 5.x.

Starting in Elasticsearch 2.3, a deprecation log became available that can help you determine if your applications are affected. Before you upgrade, check this deprecation log. If you are using deprecated queries, you will need to update your applications.

A number of other items were changed or removed in Elasticsearch 5.0 and later. For more information, check Breaking changes in 5.6.

Migrating Shield configurations

edit

In Elasticsearch 5.0 and later, the security features required to keep your Elasticsearch Add-On for Heroku clusters safe became part of X-Pack.

With the move to X-Pack, the biggest changes to security features for the Elastic Stack include the names of the security configuration options, TLS/SSL configuration, and how roles are defined. A few privileges have also been removed.

When you upgrade an Elasticsearch cluster on Elasticsearch Add-On for Heroku to version 5.x, the upgrade process to the new X-Pack security features is handled for you. As part of the upgrade process, all users, roles, and user-role mappings that exist in your Shield configuration are upgraded to use the new X-Pack security features. In addition, two users are always created on version 5.x clusters, the elastic superuser and the anonymous user. Note that in Elasticsearch versions 7.10 and higher the anonymous user is disabled by default. Check Enabling anonymous access to enable it.

If your cluster never had an active Shield configuration, or if you are not provided the password during the upgrade process, you might need to reset the password for the elastic user after upgrading to Elasticsearch 5.x. Users and applications will need to authenticate to be able to connect to your cluster.

After the upgrade is complete, you use the Kibana Management app for X-Pack to work with users and roles, which replaces the Security Editor in previous releases:

Indices originally created in older versions of Elasticsearch

edit

Indices created in Elasticsearch before version 2.0 are not compatible with version 5.x, even if you upgraded your cluster to version 2.0 or later at some point. To upgrade to Elasticsearch 5.x with these indices, you must perform some additional steps.

If you upgrade to Elasticsearch 5.x with indices that are not supported, your Elasticsearch cluster will not start. Please make sure you follow the steps in this section first.

The following steps apply to you if your current cluster was originally created in a version of Elasticsearch 2.0 or earlier, or if you are not sure what version your cluster originally used:

Check if you can upgrade to Elasticsearch 5.x directly.

For all versions of Elasticsearch: You can find out what version of Elasticsearch an index was created with by querying the index settings and looking at the value of settings.index.version.created. The first number in the value indicates the major version of Elasticsearch and any value that is below 2000000 is not readable in Elasticsearch 5.x, even if settings.index.version.upgraded shows a value that is greater than or equal to 2000000.

For example:
```
- newdata: {
    aliases: { },
  + mappings: {…},
  - settings: {
      - index: {
          refresh_interval: "-1",
          number_of_shards: "5",
        - translog: {
            durability: "async"
        },
        creation_date: "1460043613621",
        number_of_replicas: "1",
        uuid: "n6kYaMYGT0OJx8McwGAzPw",
      - version: {
          created: "2030099"
          upgraded: "5000002"
        }
    }
  },
  warmers: { }
},
```
Here is how you read the example:
- If created indicates a value that is greater than or equal to 2000000, you can upgrade to Elastisearch 5.x directly. In this example, the index was created with version 2.3.0 (created: "2030099"). You can skip the rest of the steps in this section and proceed to upgrade your cluster.
- If created indicates a value that is smaller than 2000000, you cannot upgrade to Elastisearch 5.x directly. Before you upgrade to Elasticsearch 5.x, follow the remaining steps in this section first.
  
  Examples of created values that prevent you from upgrading to Elastisearch 5.x directly:
  
  1070199
  
  Indicates an index created with Elasticsearch 1.7.1.
  
  1060299
  
  Indicates an index created with Elasticsearch 1.6.2.
For Elasticsearch 2.0 and later: You can get a human-friendly value for the version that an index was created with by appending ?human=true to your query URL.

For example, for an Elasticsearch cluster a1b2c3d4a1b2c3d4a1b2c3d4a1b2c3d4 with the index named test, the URL https://a1b2c3d4a1b2c3d4a1b2c3d4a1b2c3d4.us-east-1.aws.found.io:9243/test?human=true returns a human-friendly version with the created_string (some output has been omitted for clarity):
```
...
created_string: "2.3.2",
created: "2030299"
...
```
If you cannot upgrade to Elasticsearch 5.x directly, there are several options to get your cluster to version 5.x with minimal fuss:
- Use the migration plugin that is available for Elasticsearch 2.3.0 and later by upgrading to 2.3 first. This means that you migrate in a two-step process, first by upgrading from 1.x to 2.3 and then by upgrading from 2.3 to 5.x with the migration plugin. This process performs an online reindex.
- Use the upgrade functionality in Elasticsearch Add-On for Heroku built around the Upgrade API. This option performs a remote reindex of your indices and checks if the upgraded cluster will be able to start after upgrading.
  
  There is additional overhead when you use this option. If your cluster is already running at capacity before the upgrade, Elastic recommends that you temporarily step up to the next available cluster size before upgrading, so that the reindex operation can complete in a reasonable amount of time. You can step down to the original cluster size after upgrading to avoid further costs.
- If neither of those options work for you, you can also upgrade your indices on a cluster running Elasticsearch 2.3.x yourself with the Reindex API. For more information, check Reindex to upgrade.

A Note on Snapshots

Snapshots cannot be restored in Elasticsearch 5.x, if they contain indices created in an Elasticsearch version before 2.0. To work around this restriction, you either need to discard these snapshots or you need to open them on a cluster running Elasticsearch 2.3 before reindexing and creating new snapshots.

« Upgrade to Elasticsearch 6.x Stop routing requests »