How to

Elasticsearch 2.0.0 GA released

By
Clinton Gormley

With 2,799 pull requests by 477 committers added since the release of Elasticsearch 1.0.0, we are proud to announce the release of Elasticsearch 2.0.0 GA, based on Lucene 5.2.1.

As if that were not enough, we are also releasing version 2.0.0 of the Shield security and Watcher alerting plugins, an all new streamlined Marvel monitoring plugin which is now free to use in production, and a new open source Sense editor.

You can download Elasticsearch 2.0.0 and read about the important breaking changes in 2.0.0 here. The full changes list can be found here:

Change logs for the commercial plugins can be found here:

New in Elasticsearch

Elasticsearch 2.0.0 delivers awesome new features, such as:

Pipeline Aggregations

The ability to run aggregations such as derivatives, moving averages, and series arithmetic on the results of other aggregations. This functionality was always doable client-side, but pushing the computation into Elasticsearch makes it easier to build more powerful analytic queries, while simplifying client code considerably. It opens up the potential for predictive analytics and anomaly detection. You can read more about Pipeline Aggregations in:

Query/Filter merging

Filters are no more. All filters clauses have now become query clauses instead. When used in query context, they have an effect on relevance scoring and, when used in filter context, they simply exclude documents which don’t match, just like filters do today. This restructuring means that query execution can be automatically optimized to run in the most efficient order possible. For instance, slow queries like phrase and geo queries first execute a fast approximate phase, then trim the results with a slower exact phase. In filter context, frequently used clauses will be cached automatically whenever it makes sense to do so. You can read more in “Better query execution coming to Elasticsearch 2.0”.

Configurable store compression

Stored fields like the _source field can be compressed either with LZ4 for speed (default), or with DEFLATE for reduced index size. This is particularly useful for logging, where old indices can switch to best_compression before being optimized. You can read more in “Store compression in Lucene and Elasticsearch”.

Hardening

Elasticsearch now runs under the Java Security Manager, which marks a huge leap forward in terms of security. The Security Manager makes Elasticsearch harder to exploit and severely restricts the impact that any hacker could have on your system. Elasticsearch has also been hardened from an indexing perspective:

  • Documents are now fsynced to disk before indexing requests are acknowledged making writes durable-by-default.
  • All files are checksummed to detect corruption early.
  • All file renames are atomic to prevent any partially written files.

Finally, a much requested change from system administrators to prevent an unconfigured node from joining a public network: Elasticsearch now binds to localhost only by default, and multicast has been removed in favour of unicast.

Performance and resilience

Besides the above, there are a multitude of smaller changes both in Elasticsearch and Lucene that add up to a more stable, reliable, easy-to-configure system, for example:

  • Lower heap usage with doc-values-by-default, reduced memory usage during merges, and roaring bitsets for filter caching.
  • Structured, readable, exceptions.
  • More reliance on feedback loops instead of settings for auto-regulation.
  • A big cleanup to type mappings to make them safe, unambiguous, and reliable.
  • Cluster state diffs for faster change propagation and more stable large clusters.
  • Improved compression of norms, previously a big user of heap space.
  • Auto-throttling of merges, without needing to tweak arcane settings.
  • More fine-grained Lucene memory reporting.
  • Parent/child rewritten to take advantage of optimal query execution.
Core plugins

The officially supported core plugins now ship at the same time and with the same version number as Elasticsearch core. No longer will you need to look at a complicated version matrix to figure out which plugin version to install. Instead, the core plugins can be installed as follows:

bin/plugin install analysis-icu

New in Shield and Watcher

Our commercial plugins ship with some cool new features, like:

Shield

  • Field- and Document-level access control.
  • User impersonation.
  • Custom extendable authentication realms.
Watcher

  • Activate/Deactivate individual watches.
  • Notifications in Slack and Hipchat.

You can read more about these features in “Shield, Watcher, and Marvel 2.0.0 GA Released”.

Like the core open source plugins, our commercial plugins are now released at the same time and with the same version number as Elasticsearch core, and they can be installed as follows:

bin/plugin install license
bin/plugin install shield
bin/plugin install watcher

Marvel 2.0.0 free to use in production

The Marvel monitoring plugin has been invaluable to our customers, helping them both to diagnose problems after the fact and to spot issues while they are evolving. We have taken a good hard look at what can be improved and have rewritten Marvel from scratch:

  • The Marvel UI is now built on top of the all new Kibana platform.
  • Dashboards have been streamlined to show the most important metrics, making problems easier to spot.
  • Marvel now supports monitoring of multiple clusters from a single installation, as a commercial feature.

And the best part - Marvel is now free to use in production for all Elasticsearch users. A license is required, but is available to all users free of charge. If you require multi-cluster monitoring support, that is a commercial feature.

You can read more about Marvel in "Shield, Watcher, and Marvel 2.0.0 GA Released".

Open source Sense editor

Sense, the browser-based Elasticsearch request and DSL editor, is now available to all as an open source app built on top of the Kibana platform. This new release adds some great new features:

  • Paste multiple cURL requests to convert to Sense syntax.
  • Copy multiple Sense requests in cURL syntax.
  • Execute multiple requests in one go.
  • Autocompletion database updated to support Elasticsearch 2.0.

Sense can be installed as a Kibana app as:

./bin/kibana plugin --install elastic/sense

You can read more about Sense in "The Story of Sense - Announcing Sense 2.0.0-beta1".

Elasticsearch Migration Plugin

The Elasticsearch Migration Plugin is the best starting point if you are upgrading from Elasticsearch 1.x to 2.0. It installs as a site plugin in any 1.x Elasticsearch cluster, and will detect issues that need to be resolved before upgrading such as ancient Lucene 3 indices and problematic mappings (see “The Great Mapping Refactoring”) that will no longer work in Elasticsearch 2.0.0.

You can find the instructions for this plugin in the Elasticsearch Migration repository.

Conclusion

Please download Elasticsearch 2.0.0, try it out, and let us know what you think on Twitter (@elastic) or in our forum. You can report any problems on the GitHub issues page.