Elasticsearch 5.6.0 released

Elasticsearch 5.6.0 is the Upgrade Release. If you want to be able to do a rolling upgrade from 5.x to 6.x, you will need to upgrade your cluster to 5.6.0 or higher first. Besides being the 5.x release that will allow rolling upgrades to 6.x, Elasticsearch 5.6.0 comes with the latest deprecation logging to warn you about deprecated features that you are using that you will need to replace before migrating to 6.0. Even if you’re planning on doing a full cluster restart to get to 6.x, we highly recommend first going to 5.6.0 or higher in order to look at the deprecation logs to ensure that your application is ready for the upgrade.

On top of that, after removing site plugins, we revisited how we wanted to provide additional tooling for upgrade assistance as the old "migration assistant" was, in fact, a site plugin such that the approach would no longer work.  We wanted to bundle both the UI and any new APIs for the assistance together so that you could experience as easy of a migration path as possible.  We also had requests in the past for a CLI-compatible version of the old migration assistant.  We thought long and hard about this and decided the best path forward was to put the old migration assistant functionality (and even more!) inside of X-Pack, since this can bundle UI (Kibana) and Elasticsearch extensions.  However, we wanted to make sure that this functionality was available in a completely free tier of X-Pack's licensing so that you wouldn't have to pay for this type of basic upgrade assistance.

With that, on to the new Upgrade Assistant features!  Kibana X-Pack now includes a “Cluster Checkup” tool, to warn you about changes that need to be made in your cluster before upgrading.  It also includes a “Reindex Helper” to prepare your indices for 6.x compatibility by upgrading your Security, Watcher, and Kibana indices, and reindexing indices created in 2.x. These Kibana UIs are backed by new Migration APIs in Elastisearch X-Pack.  If you're operating in a pure command-line environment and/or can't install Kibana for some reason, you can use these APIs directly.

Again, these X-Pack Migration tools require either a Trial license (auto-generated the first time that you use X-Pack), or a free Basic license.

NOTE: TLS/SSL encryption between nodes is required in 6.0 if you use X-Pack Security. If you are currently using X-Pack Security without TLS, you will not be able to do a rolling upgrade to 6.0. A full cluster restart is required to enable TLS, either in 5.x or in the transition to 6.x.

The wait is finally over! In Elasticsearch 5.0, we released the low-level Java REST client, which provides the same resilient connections as our REST clients for other languages, but was difficult to use for a Java programmer as it only accepted and returned JSON blobs. The new high-level REST client is built on top of the low-level client, but it accepts request objects and returns response objects for the most important APIs: info, get, index, delete, update, bulk, search, search-scroll, and clear-scroll. These builders and helpers make the client much easier to use in an IDE. Other requests can still be handled with the low-level client.

This finally puts Java at the same level as other languages. The high-level HTTP client will become the official way for Java applications to communicate with Elasticsearch, replacing the Transport Client, which will be removed in Elasticsearch 7.0.

The biggest change coming in 6.0 is the (beginning of the) removal of mapping types. This 5.6.0 release includes the join datatype — the new way to specify parent/child relationships without needing types. While your existing 5.x indices will continue to work as before in 6.x, we wanted to make the join datatype available in 5.x to help you prepare your application for the upgrade to 6.0.

A search against an index pattern like logstash-* can fan out to a huge number of shards. Usually, queries like these include a date range filter which means that the majority of shards won’t contain any matching documents. We already include an optimization to abort search requests on these shards early, before any real work is done, but this is not enough. Imagine a multi-search request containing 10 search requests, each of which target 2,000 shards. That’s 20,000 shard-level search requests which are added to the search threadpool queue. This could easily result in rejections, even though the majority of these requests are very quick.

Previously, Kibana used the _field_stats API with a date range filter to figure out which indices might contain matching documents, and then ran the search request against only those indices. We wanted to remove this API because it was much heavier than users expected and open to abuse. Instead, a search request now has a light shard prefiltering phase which is triggered if a search request targets at least 128 shards (by default). These prefilter requests are not added to the search queue and so cannot be rejected because the queue is full. The prefilter request rewrites the query at the shard level and determines whether the query has any chance of matching any documents at all. The full search request is then sent only to those shards which have a chance of matching.

But what if the user actually does want to search all 2,000 shards, or searches all indices by mistake? These wide-ranging requests should not overwhelm the cluster nor get in the way of search requests from other users. In order to solve this, we introduced the max_concurrent_shard_requests parameter whose default value depends on the number of nodes in the cluster, but which has a fixed upper limit of 256. This may make a single search request that targets many shards slower, but it makes for fairer concurrent searches by many users.

Please download Elasticsearch 5.6.0, try it out, and let us know what you think on Twitter (@elastic) or in our forum. You can report any problems on the GitHub issues page.