Product release

Elasticsearch 5.0.0-alpha5 released

By
Clinton Gormley

Today we are excited to announce the release of Elasticsearch 5.0.0-alpha5 based on Lucene 6.1.0. This is the fifth in a series of pre-5.0.0 releases designed to let you test out your application with the features and changes coming in 5.0.0, and to give us feedback about any problems that you encounter.

Open a bug report today and become an Elastic Pioneer.

IMPORTANT: This is an alpha release and is intended for testing purposes only. Indices created in this version will not be compatible with Elasticsearch 5.0.0 GA.

DO NOT DEPLOY IN PRODUCTION

Development of Elasticsearch 5.0.0 is nearing completion. This release contains more bug fixes and enhancements than it does new features (all of which you can read about in the release notes linked above), but there are a few gems worth mentioning.

Also take a look at the release announcements for Elasticsearch 5.0.0-alpha4, Elasticsearch 5.0.0-alpha3, Elasticsearch 5.0.0-alpha2, and Elasticsearch 5.0.0-alpha1 to read about features like:

  • Java REST client
  • Rollover indexing
  • Wait for refresh
  • Ingest Node
  • Painless Scripting
  • Instant Aggregations
  • Text/Keyword fields replacing String
  • Completion Suggester v2
  • Settings Validation
  • Safety in Production
  • Resiliency Improvements
  • Percolate Query
  • Deleted Index Tombstones
  • Dots in field names
  • Cluster allocation explain API

Known networking bug in 5.0.0-alpha5

We have discovered is a major bug in the new Netty4 implementation in this release which affects any REST requests greater than 1024 bytes in size, and which will generate an exception similar to the following:

[WARN ][http.netty4] [wtOV9Vb] caught exception while handling client http traffic, closing connection [id: 0x1320b717, L:/0:0:0:0:0:0:0:1:9200 - R:/0:0:0:0:0:0:0:1:54732]
java.lang.UnsupportedOperationException: unsupported message type: DefaultFullHttpResponse (expected: ByteBuf, FileRegion)

This is due to incorrect handling of the Expect HTTP header, and it can be worked around in one of three ways:

  • Use a client which does not add Expect headers (including the official clients).
  • Pass a blank Except header, e.g. 
    curl -H 'Expect:' ...
  • Use Netty3 for the HTTP layer by passing the following setting at startup: 
    ./bin/elasticsearch -Ehttp.type=netty3

Migration Helper

The Elasticsearch Migration Helper is a site plugin designed to help you to prepare for your migration from Elasticsearch 2.3.x to Elasticsearch 5.0. It comes with three tools:

Cluster Checkup
Runs a series of checks on your cluster, nodes, and indices and alerts you to any known problems that need to be rectified before upgrading.
Reindex Helper
Indices created before v2.0.0 need to be reindexed before they can be used in Elasticsearch 5.x. The reindex helper upgrades old indices at the click of a button.
Deprecation Logging
Elasticsearch comes with a deprecation logger which will log a message whenever deprecated functionality is used. This tool enables or disables deprecation logging on your cluster.

Instruction for install the Elasticsearch migration helper.

Index creation friendly to sysadmins

In previous versions of Elasticsearch, when you issue a create-index request you receive a response as soon as the index is added to the cluster state, even though the shards of the index have not yet been allocated. An index with unallocated primary shards turns the cluster health red, which rings an alarm bell for sysadmins. As of this release, Elasticsearch now waits for the primary shards to be allocated before responding. This means that your index is ready to use as soon as the create-index request returns. On top of that, even dynamically created indices will no longer turn the cluster red (however briefly) unless there is a real problem which means that shards cannot be allocated. As part of this change, the write_consistency parameter has been renamed to wait_for_active_shards, which more clearly expresses the behaviour of this parameter.

Netty4

Netty is the library that we use for networking. Changes to the design in Netty 4.0 made it difficult for us to upgrade, but those issues have been resolved in Netty 4.1. Upgrading the networking layer is a big and potentially risky change, so in this version we ship with a Netty4 and a Netty3 module. Netty4 is used by default but, in case we discover any problems, you can switch back to Netty3 by updating the transport.type and http.type settings to netty3.

Aggregation improvements

The histogram aggregation now supports fractional buckets and handles correct rounding of negative buckets. On top of improvements added in alpha4 to reduce the number of buckets created by the terms aggregation where possible, this release reduces the default number of buckets created per shard to make the terms agg lighter. Of course, multi-level aggregations on high cardinality fields can still result in combinatorial explosion with trillions of buckets, so the request circuit breaker has been extended to protect your cluster from this abuse.

Security improvements in X-Pack

Getting started with security has historically been hard, especially when wrangling openssl to generate self-signed certificates. X-Pack now ships with a command line tool which will generate certificates (and optionally a certificate authority) for you in the simplest way possible. We have also simplified document level security by allowing metadata associated with roles (eg access_level) to be plugged into a DLS query. 

Other Notable Changes

  • The new scaled_float field datatype allows storing a floating point number as a long value, which Lucene can compress more efficiently.
  • Numeric fields, indexed as block KD trees, have received further improvements to reduce disk storage, and to speed up both indexing and search performance.
  • Custom tokenizers, token filters, and character filters can be specified inline in the _analyze API.
  • An unrecoverable exception (such as out-of-memory) will now cause a node to die with dignity, instead of limping on in an undefined state.
  • The Java REST client now supports blocking and asynchronous requests.
  • Nodes now persist their IDs across restarts, making it easier to monitor and debug problems.

Conclusion

Please download Elasticsearch 5.0.0-alpha5, try it out, and let us know what you think on Twitter (@elastic) or in our forum. You can report any problems on the GitHub issues page.