Product release

Elasticsearch 1.7.0 and 1.6.1 released

By
Clinton Gormley

Today, we are pleased to announce the release of Elasticsearch 1.7.0 based on Lucene 4.10.4, and the bugfix release of Elasticsearch 1.6.1. These releases contain a security fix and all users are advised to upgrade.

You can download them and read the full changes list here:

The 1.7.0 release will be the last feature release in the 1.x series. All future features will go into Elasticsearch 2.0 or later.

Elasticsearch 1.7.0 is a small release, but contains two important security fixes and two important features which will improve cluster stability and recovery:

Security fixes

Both Elasticsearch 1.6.1 and 1.7.0 include the following two security fixes:

Remote code execution vulnerability

Elasticsearch versions prior to 1.6.1 are vulnerable to an engineered attack on its transport protocol (used for communication between nodes and Java clients) that enables remote code execution. This issue is related to the Groovy announcement in CVE-2015-3253.

Deployments are vulnerable even when Groovy dynamic scripting is disabled. Users that do not want to upgrade can address the vulnerability by securing the transport protocol port (default 9300) to allow access by only trusted agents.

We have been assigned CVE-2015-5377 for this issue.

Directory traversal vulnerability

Elasticsearch versions from 1.0.0 to 1.6.0 are vulnerable to a directory traversal attack that allows an attacker to retrieve files that are readable by the Elasticsearch JVM process. Users that do not wish to upgrade can use a firewall, reverse proxy, or Shield to prevent Snapshot-Restore API calls from untrusted sources.

We have been assigned CVE-2015-5531 for this issue.

Delayed shard allocation

Elasticsearch 1.6.0 introduced Synced Flushing, which dramatically speeds up the recovery of idle shards when restarting a node. This change, however, only works well in controlled environments when shard allocation is disabled. It doesn’t help when a node is temporarily disconnected from the cluster or undergoes an unexpected reboot.

This scenario plays out as follows:

  • A node shuts down unexpectedly.
  • The master reassigns its shards to other nodes.
  • Each shard is copied over the network to their new locations.
  • Meanwhile, the missing node rejoins the cluster.
  • The master redistributes shards to take advantage of the “new” node. Potentially none of the existing shards on the new node are reused.

Even though we throttle concurrent recoveries both at the node level and at the cluster level, this “shard-shuffle” can still put a lot of extra load on the cluster, which could be entirely avoided by simply waiting for the missing node to rejoin.

Wait for it!

Elasticsearch 1.7.0 adds the index.unassigned.node_left.delayed_timeout setting, which defaults to 1 minute. This means that, when a node leaves the cluster, the master will wait for one minute before reallocating these shards to other nodes. If the node returns during that one minute period, the master will reallocate to it the shards that it already has locally.

Why one minute?

We chose one minute because it gives enough time for a node to shut down, restart, and rejoin the cluster, but it still means that reallocation happens if the node doesn’t return. A good default is hard to choose. You may choose to reduce or increase this setting depending on how closely you monitor your cluster.

A default value for this setting can be configured in the config/elasticsearch.yml file for new indices, but it can be updated dynamically on live indices using the update index settings API.

Your feedback of real world experiences will help us to refine this default.

Prioritization of index recovery

The second important feature in 1.7.0 is the ability to prioritize the order in which indices are recovered after, for example, a full cluster restart.

Imagine that a power failure takes your whole logging cluster down. The cluster comes back up and has to recover 500 indices, but 499 of them hold old data and the only one you are interested in is the 500th. You cannot resume indexing until the most recently created index has recovered.

Up until now, indices were recovered in random order and you just had to wait until the one you’re interested in comes back online. In 1.7.0, indices are recovered in order of priority, where the priority is determined by the following properties:

  • The optional index.priority setting (higher is better).
  • The index creation date (higher is better).
  • The index name (higher is better).

Without you having to make any changes to your existing cluster, more recently created indices will be recovered before older indices. If you need to bump up the priority of an older index, set the dynamic index.priority setting to a number higher than zero:

PUT important_index/_settings
{
  "index.priority": 5
}

This setting can be updated on a live index, or even during recovery.

Conclusion

Please download Elasticsearch 1.7.0, try it out, and let us know what you think on Twitter (@elastic) or on our forum. You can report any problems on the GitHub issues page.