Elasticsearch 6.7.0 released
We are pleased to announce the release of Elasticsearch 6.7.0, based on Lucene 7.7.0. This is the latest stable release, and is already available for deployment via our Elasticsearch Service on Elastic Cloud.
Latest stable release in 6.x:
You can read about all the changes in the release notes linked above, but here are our highlight:
If it’s a 3-letter acronym in Elasticsearch, odds are it’s now Generally Available (GA) in 6.7!
CCR (Cross Cluster Replication) is now GA!
We introduced Cross Cluster Replication (CCR) as a beta feature in Elasticsearch 6.5. CCR was one of the most heavily requested features for Elasticsearch. We’re excited to announce CCR is now generally available and ready for production use in Elasticsearch 6.7! CCR has a variety of use cases, including cross-datacenter and cross-region replication, replicating data to get closer to the application server and user, and maintaining a centralized reporting cluster replicated from a large number of smaller clusters.
In addition to maturing to a GA feature, there were a number of important technical advancements in CCR for 6.7. Previous versions of CCR required replication to start on new indices only: existing indices could not be replicated. CCR can now start replicating existing indices that have soft deletes enabled in 6.7. We introduced new technology to prevent a follower index from falling fatally far behind its leader index. We’ve added a management UI in Kibana for configuring remote clusters, indices to replicate, and index naming patterns for automatic replication (e.g. for replicating metricbeat-* indices). We’ve also added a monitoring UI for insight into CCR progress and alerting on errors. Check out the Getting started with cross cluster replication guide, or visit the reference documentation to learn more. Cross cluster replication is a platinum level feature, and is available through a 30-day trial license that can be activated either via an API or within Kibana.
SQL, including JDBC and ODBC drivers, is now GA!
The SQL interface to Elasticsearch is now GA. Introduced in 6.3 as an alpha release, the SQL interface allows developers and data scientists familiar with SQL to use the speed, scalability, and full-text power of Elasticsearch that others know and love. It also allows BI tools using SQL to easily access data in Elasticsearch. In addition to approving SQL access as a GA feature in Elasticsearch, we’ve designated our JDBC and ODBC drivers as GA. There are four methods to access Elasticsearch SQL: through the Elasticsearch REST endpoints, the Elasticsearch SQL command line interface, the JDBC driver, and the ODBC driver.
Not only is SQL available in Elasticsearch 6.7, we’ve introduced a number of new features. We’ve added a proper SQL DATE type, introduced the ability to reference an alias within a where clause (“SELECT age AS a FROM profile WHERE a > 21”), added sorting of groups by aggregates ("SELECT age, COUNT(*) FROM test GROUP BY age ORDER BY COUNT(*) DESC LIMIT 100”), and added FIRST/LAST aggregate functions, to name a few.
To explore this new feature in depth, see the dedicated documentation section covering everything from usage to syntax and drivers for new users and experts alike. The SQL interface in Elasticsearch is free to use and part of our default distribution, the ODBC and JDBC drivers are part of a platinum subscription.
ILM (Index Lifecycle Management) is now GA!
Index Lifecycle Management (ILM) was released as a beta feature in Elasticsearch 6.6. We’ve officially moved ILM out of beta and into GA, ready for production usage! ILM makes it easy to manage the lifecycle of data in Elasticsearch, including how data progresses between hot, warm, cold, and deletion phases. Specific rules regarding how data moves through these phases can be created via APIs in Elasticsearch, or a beautiful management UI in Kibana.
In Elasticsearch 6.7, ILM can now manage frozen indices. Frozen indices are valuable for long term data storage in Elasticsearch, and require a smaller amount of memory (heap) in relation to the amount of data managed by a node. In 6.7, frozen indices can now be frozen as part of the cold phase in ILM. In addition, ILM now works directly with Cross Cluster Replication (CCR), which also GA’d in the Elasticsearch 6.7 release. The potential actions available in each ILM phase can be found in the documentation. ILM is free to use and part of the default distribution of Elasticsearch.
Get Ready to Upgrade to Elasticsearch 7.0
Elasticsearch 6.7 is the Upgrade Release, meaning if you want to do a rolling upgrade from 6.x to 7.x, you will need to upgrade your cluster to 6.7.0 or higher first. Besides being the only 6.x release that will allow rolling upgrades to 7.x, Elasticsearch 6.7.0 comes with the latest deprecation logging to warn you about deprecated features you are using that will need to be replaced before migrating to 7.0. Even if you’re planning to do a full cluster restart to get to 7.x, we highly recommend first going to 6.7.0 or higher to look at the deprecation logs and ensure that your application is ready for the upgrade.
Guided upgrade with the Upgrade Assistant
We release an Upgrade Assistant before every major version, and Elasticsearch 7.0 is coming. The Upgrade Assistant is great for receiving a cluster checkup before an upgrade, such as deprecation warnings or a list of indices that need to be upgraded or reindexed.
We’re excited to announce major improvements to the Upgrade Assistant for the Elasticsearch 6.7 to 7.x upgrade path! We’ve greatly enhanced the reindex functionality within the Upgrade Assistant. When a reindex operation is initiated through the Upgrade Assistant, the process is now managed in the background. Users can navigate away from the page and get an updated status for the reindex operation when they return. We’ve also added functionality to ensure a reindex operation has completed successfully, and if not, revert any changes to get back to the initial state.
Upgrade APIs
Although there is a UI for managing an Elasticsearch upgrade, there are scenarios where the upgrade must be programmatically managed through a set of APIs. For instance, the Deprecation Info APIs can be used to learn about deprecated features that may have been removed in Elasticsearch 7.0. We also enhanced the Upgrade Assistant APIs to initiate and track reindex status. For instance, the Upgrade Assistant Reindex API can be used to initiate a reindex request, cancel it, and then check the status on the cancel request.
Reindex Improvements
Sometimes your upgrade path may require you to reindex data. To make it easier to reindex your data from a remote cluster, we’ve made two improvements. You are now able to apply custom SSL parameters when reindexing, which makes it possible to trust your own CA without needing to modify the JVM’s default trusted certificates. We also added support for reindexing from IPv6 URLs.
Security Improvements in Elasticsearch
Pluggable Authorization Engines
Elasticsearch has had pluggable authentication for a while now, which has allowed our customers to tie into complex and custom authentication mechanisms which may be part of their application or enterprise. In 6.7, Elasticsearch now supports a similar model for custom authorization to allow access to specific documents in Elasticsearch with custom rules. This will allow organizations to build complex and custom authorization mechanisms. To learn about building your own custom authorization engine for Elasticsearch, visit our reference documentation for customizing roles and authorization.
Bundling the popular geo-ip and user-agent plugins by default
The geo-ip plugin for Elasticsearch is a popular plugin that resolves IPv4 and IPv6 address to an approximate physical location. Resolving IP address to location is extremely valuable to a number of use cases, especially logging. The user-agent plugin extracts information from the ‘User-Agent’ header a browser sends with its requests. Both plugins are now bundled with Elasticsearch by default, alleviating additional steps to download and install both of these very popular Elasticsearch plugins.
Improvements to Elasticsearch Index Management
We’ve made a number of improvements to the index management UI for Elasticsearch for 6.7. Rollup indices, follower indices, and frozen indices are now marked with a badge, to easily distinguish them from one another. Building on the support for frozen index integration in ILM, we’ve also made it easy to freeze and unfreeze indices within the Index Management UI. To use the index management UI for Elasticsearch 6.7, please download Kibana 6.7, click the gear icon (Management) within the left hand nav, and navigate to Elasticsearch Index Management.
Conclusion
Please download Elasticsearch 6.7.0, try it out, and let us know what you think on Twitter (@elastic) or in our forums. You can report any bugs or feature requests on the GitHub issues page.