Elastic Stack 7.0.0 released

7.0 is here! This release represents more than 10,000 pull requests from 861 committers, so first, a big thank you to our employees and community.

If you’d like to hear about the release from the people behind the code, we’ll be hosting a live virtual launch event on April 25, 2019, at 8 am PDT. Join us for 7.0 demos, an AMA with Elastic engineers from around the globe, and more.

Elastic Stack 7.0 is immediately available for download, or you can spin up fully managed deployments on the Elasticsearch Service on Elastic Cloud — the only hosted solution to offer new Elastic Stack versions the day they launch.

With so much goodness in 7.0, it’s hard to know where to start, so let’s dive right in.

Kibana 7.0: new design & navigation … and dark mode!

With the Kibana 7.0 design, we decided to place the focus on the content, so you will notice that the UI has a lighter, more minimal feel throughout. The most prominent change is a switch to a new global navigation, which introduces a constant header to switch Kibana spaces, display breadcrumbs, and initiate user actions like change password or logout. To achieve this, and increase consistency, we created the Elastic UI Framework. Over the last year, we converted nearly all of Kibana to use these components. With these components, and a herculean effort from our design and engineering teams, we also made dramatic simplifications to how styles and style sheets are applied.

The increased consistency and style sheet improvements enabled us to check off what felt like one of the biggest feature requests in Kibana history — dark mode across all of Kibana. As another benefit of these changes, Kibana dashboards now have a responsive design, which is the first step in dramatically improving usability on mobile devices.

A new era for cluster coordination in Elasticsearch

Since the beginning, we have focused on making Elasticsearch easy to scale and resilient to catastrophic failures. To support these requirements, we have taken multiple approaches, from making individual nodes more scalable and reliable, to continuous improvement to our cluster coordination layer, known as Zen Discovery. With 7.0, we’re introducing big improvements in both areas yet again.

There is a completely new cluster coordination layer for Elasticsearch, which is faster, safer, and easier to use. To achieve this, we started by focusing on the theoretical correctness of our new distributed consensus algorithm using formal models to validate the design. While there are well-known consensus algorithms, like Paxos, Raft, Zab and Viewstamped Replication (VR), the demands of an Elasticsearch cluster require higher throughput for cluster changes, support for easily growing or shrinking a cluster, and a seamless rolling upgrade strategy to allow 6.7 clusters to do a rolling upgrade to 7.0, features that these reference algorithms couldn’t provide. The new cluster coordination layer also includes a number of changes that reduce the likelihood of human error and provides clearer choices when recovering from catastrophic failure. It’s not easy to improve reliability, performance and user experience all at once, especially in such a central component. We’re proud of the new cluster coordination layer, and the process we undertook to get here. To learn more, read the blog.

Individual nodes in Elasticsearch are built with resiliency in mind. If you send too many requests to a node or your requests are too large, the node will push back. We achieve this with circuit breakers in Elasticsearch, which determine that the node wouldn’t be able to handle a given request, and immediately respond by asking the client to retry, perhaps on a different node. For nodes with smaller JVM heap sizes, which are becoming more common as users move to a cluster-per-tenant model rather than a massive multitenant cluster, this is even more important. In 7.0, we’re introducing the real memory circuit breaker, which much more accurately detects unserviceable requests, and prevents them from making an individual node unstable. Read the blog to learn more about how this change improves overall node and cluster reliability.

Giving relevance and speed a boost across use cases

Relevance and speed are the cornerstones of a good search experience. And Elasticsearch 7.0 introduces several foundational features that improve both.

• Faster top k queries: In many search use cases, quickly seeing the top k (say 20) results on a query matters much more to the user than the exact hit count (i.e., total number of results matching the query). For example, if someone is searching for a product on an e-commerce website they are much more interested in the 10 most relevant results, rather than the other 120,897 results that matched their search query. Elasticsearch 7.0 (and Lucene 8.0) implements a new algorithm (Block-Max WAND) that provides a huge speed boost when retrieving top hits.
• Intervals queries: Some search use cases, for example, legal and patent search, introduce the need to find records in which words or phrases are within a certain distance from each other. Intervals queries in Elasticsearch 7.0 introduce a brand new way of structuring such queries, and are significantly simpler to use and define compared to the previous method (span queries). Intervals queries are also much more resilient to edge cases compared to span queries.
• Function score 2.0: Custom scoring is the bread and butter of advanced search use cases, where one would want finer control on relevancy and results ranking. Elasticsearch has provided the ability to do this since its early days. 7.0 introduces the next generation of function score capability that provides a simpler, modular, and more flexible way to generate a ranking score per record. The new modular structure allows users to mix and match a set of arithmetic and distance functions to construct arbitrary function score calculations, giving them more control over how results are scored and ranked.

Smooth zoom in Elastic Maps with geotile grid

Over the years, our support for geo data has continuously improved — from the early days when geo support was first added to Elasticsearch, to introducing the Bkd-Tree data structure to Lucene and using it to improve geoshape query performance by over 25x, to the Elastic Maps service that powers the global basemap in Kibana.

With 7.0, we continue this investment, introducing a new aggregation in Elasticsearch to handle (geo) map tiles in a way that allows a user to zoom in and out on the map without any change to the shape of the result data. The new geotilegrid aggregation groups geopoints into buckets that represent cells in a grid, with each cell correlating with a tile in a map. Prior to this change, the fringes of the shape could slightly change with the change in zoom level, because the rectangular tiles would change orientation at different zoom levels. Elastic Maps in 7.0 is already using this new aggregation to ensure that your view stays stable as you zoom in and out. This level of accuracy is important, whether you're protecting your network from attackers, investigating slow application response times in specific locations, or tracking your brother hiking the Pacific Crest Trail.

Strengthening time series use cases with nanosecond precision support

Whether it’s infrastructure metrics, system audit logs, network traffic, or a rover on Mars, time series data is central to how many people use the Elastic Stack. The ability to precisely order and correlate events across multiple systems and services is key.

Up until now, Elasticsearch only stored timestamps with millisecond precision. 7.0 adds a few zeroes and brings nanosecond precision, which allows users with high-frequency data collection needs the precision required to accurately store and sequence this data. The change was made possible by migrating from the historical JODA library to the official Java time API in JDK 8.