Elasticsearch 5.2.0 released

While numeric and date fields index a discrete number or point in time, numeric and date range fields allow you to index numeric and date ranges, such as Friday 27 January 2017, between 6pm and 8pm. Range queries can be used to search for ranges which overlap, are completely contained or which contain, or which do not overlap at all. This allows you to answer questions like "What entertainment is available on Thursday evening?", a query which was previously very difficult to construct.

See Numeric and Date Ranges in Elasticsearch: Just Another Brick in the Wall for more information.

PagerDuty wakes you up at 3AM with a red cluster. Which API do you reach for to figure out why shards aren’t being allocated? Previously, you had to consult several APIs to put together the complete picture. Now, the answer is simple: the cluster allocation explain API. This API can tell you whether a shard has failed to allocate or is just waiting its turn, and why allocation failed, whether it be a corrupt shard, full disks, or bad settings. It can also tell you why a shard is assigned to a particular node, perhaps when you have tried to force relocation.

See RED Elasticsearch Cluster? Panic no longer for more information.

In 5.0.0, we separated the string field type into text (for analyzed full text) and keyword (for not-analyzed string identifiers). Fields of type text can be analyzed into individual tokens for full text search, while keywords support doc_values, used for search, aggregations, sorting, and in scripts. However, there are times when you need some of the power of the analysis chain to normalise keywords, such as lower-casing email addresses or zip codes.

This release brings normalizers to keyword fields. Normalizers are similar to analyzers except that they may only emit a single token. As a consequence, they do not have a tokenizer and only accept a subset of the available char filters and token filters. Only the filters that work on a per-character basis are allowed. For instance a lowercasing filter would be allowed, but not a stemming filter, which needs to look at the keyword as a whole.

The terms aggregations returns the top 10 terms by default. We are often asked "But how do I return ALL terms?". The answer up until now is "You don’t". It was simply too memory intensive to collect all the terms of a high cardinality field from all the nodes in the cluster and to reduce them to a single result set. It also defeated the purpose of the terms agg, which was designed to return the top-N counts from huge datasets at speed.

However, as we’ve seen many times over, users surprise us with the problems that they solve with Elasticsearch. A frequent request was to be able to return all terms, even if the response is not instantaneous. You can now break your terms down into partitions — the more unique terms you have the more partitions you will need. For instance, you could choose to use 20 partitions and run 20 search requests, each one requesting a single partition.

See Filtering Values with Partitions for more info.

Please download Elasticsearch 5.2.0, try it out, and let us know what you think on Twitter (@elastic) or in our forum. You can report any problems on the GitHub issues page.