Elastic Stack 5.0.0 Released
In February of 2016, following Elastic{ON} 16, I wrote a post titled Heya, Elastic Stack and X-Pack. Today, after almost a year of substantial effort, including 5 Alphas, 1 Beta, and 1 Release Candidate we are pleased to announce the GA release of the Elastic Stack.
And, importantly, it is available – today – on Elastic Cloud. If you want hosted Elasticsearch and Kibana there is no other place to start with the most recent code. We are committed to making Elastic Cloud the best place to run hosted Elasticsearch. In fact, we even made the Release Candidate available on cloud for testing purposes.
Our team is celebrating today. I hope you join us.
The GA release is available today. Join the Elastic Team for a live virtual event on November 3 to learn more about the release and ask the creators questions (AMA style). Register now!
Before exploring the release in detail, I want to take the opportunity to reflect on what has brought us to this point.
Our Community
During the recent Elastic{ON} Tour, I have begun each session discussing a brief history of the last several years. This session culminates in the announcement that we have reached a combined 75 Million downloads. When I first began the project, I hoped for widespread adoption. But the passion and fervor of our community continues to delight and amaze me.
Pioneer Program
With that in mind, I want to share the results of the Pioneer Program. The program began with a simple premise. Your usage of the Elastic Stack is of the utmost importance in informing our development as well as ensuring we release the highest quality product available. I am pleased to say that the community has filed 146 issues since the first Alpha release in April.
Our community is one of our most valued assets at Elastic. In fact, one of the most discussed changes in this release was the name “Elastic Stack”.
The Elastic Stack
Last year, we brought the Packetbeat team on board, and Beats was born. This open source platform for building lightweight data shippers for log files, infrastructure metrics, network packets, and more, made it easier than ever to send data to Elasticsearch and Logstash. While we love how many of you who have adopted the abbreviation ELK for our stack, with Beats, we just couldn’t figure out how to make the “B” work with the E-L-K combination.
But Elastic Stack is more than just a name. When we began this release cycle we committed to developing, building, testing, and releasing the entirety of the Stack together. This is important, internally, to ensure compatibility. And, for you, it helps speed deployment, decrease version confusion, and make it easier for developers to add capabilities across the entirety of the Elastic Stack.
A Feature Tour
When I began this post, I intended to provide an overview of key features in each product. But, it was hard to know where to begin and where to stop. Each of our team and tech leads have created a post that discusses the features specific to their product. And there is no one better suited to tell the story than them. I am, particularly, excited about a few items but rather than enumerate in detail, I will provide a brief overview and encourage you to read the detail posts for each product.
- Ingest Node - Ingest Node is an Elasticsearch node type enabling some data enrichment capabilities like grok, geoip, date, and other basic event manipulation options at index (or re-index) time. Pipelines are constructed with processors, and accessed through the REST API by suffixing a query parameter “?pipeline=x”. The ability to add pre-processing to documents, natively in Elasticsearch, prior to indexing allows for a variety of creative ingest deployments. This doesn’t replace Logstash. This doesn’t remove the need for Beats, this just allows greater flexibility in designing your ingest architecture.
- Elasticsearch Performance - Benchmarks tend to have an agenda…especially competitive benchmarks. With that in mind, we have spent substantial effort comparing 5.0.0 to prior releases. This data is available to you. This data is what we inspect when we want to ensure that we are doing the right things with performance and we are doing so in public to work towards preventing the secrecy, and doubt, that are associated with benchmark numbers. In fact, not only are the results available but we also document our hardware configuration, we have open sourced the tooling (called Rally) and the benchmarks themselves (Rally-Tracks).
- Metricbeat - Metricbeat replaces Topbeat as the primary tool for collecting metrics in the Elastic stack. Like Topbeat, Metricbeat collects “top” like statistics about host and per process resources (CPU, memory, disk, network). Unlike Topbeat, Metricbeat also collects metrics from systems such as Apache, HAProxy, MongoDB, MySQL, Nginx, PostgreSQL, Redis, or Zookeeper, with more to come in the near future.
- Logstash Monitoring APIs - A new monitoring feature provides runtime visibility into the Logstash pipeline and its plugins. This component collects various kinds of operational metrics while Logstash processes your data, and all of this information can be queried using simple APIs.
- Timelion - After being introduced as a {Re}search project, Timelion is now natively available in Kibana core. Timelion provides a query DSL and visualizations that let you explore your data over time.
This is but a sample, I’ve left out BKD trees, scaled_float and half_float, the immense effort put into Elasticsearch Resiliency, the eye-meltingly beautiful redesign of Kibana (we never knew how much we hated borders until we removed them), Kafka output in Beats, and so much more.
This is a massive release. Reading the individual posts is a must to begin to understand the scope of improvement.
X-Pack
At Elastic we loved extensions. So much so that we built them and gave them interesting names. Shield, Marvel, and Watcher all described individual closed source features that didn’t take away for open source capability but were additive for our customers. Unfortunately, as the range of these features grew to include Graph and Reporting, the install process became difficult and, at times, quite confusing.
Say Heya to X-Pack!
One pack that adds security, alerting, monitoring & management, reporting, and graph capabilities to the Elastic Stack. Our engineering process for 5.0 wasn’t limited to the Elastic Stack, but we’ve also extended X-Pack by adding:
- Management & Monitoring UIs to Kibana
- Security UIs to Kibana for creating both users and roles
- Greatly simplified the installation process
X-Pack is available to trial and has both commercial and free (Basic) license options. We are particularly excited to make some X-Pack features available for free and details are available on our Subscriptions page.
In Closing
I am in awe of the effort that went into this release, the involvement from our community and customers, and the groundwork that this sets for future releases. As always, the best way to understand a release is to experience it.