Elasticsearch 6.3.0 Released
Editor's Note (August 3, 2021): This post uses deprecated features. Please reference the map custom regions with reverse geocoding documentation for current instructions.
Today we are pleased to announce the release of Elasticsearch 6.3.0, based on Lucene 7.3.0. This is the latest stable release, and is already available for deployment via our Elasticsearch Service on Elastic Cloud.
Latest stable release in 6.x:
You can read about all the changes in the release notes linked above, but there are a few changes which are worth highlighting:
SQL
Have you (or someone you know) ever:
- Said “I know how to do this thing in a SQL statement -- how do I do the same thing in Elasticsearch?”
- Tried to build out full-text search with tokenization, stemming, synonyms, relevance sorting on top of a SQL engine like a relational database?
- Tried to scale out a traditional database to billions of rows?
- Tried to connect a 3rd party tool like a BI system to Elasticsearch?
These are all things which we hope we can make inroads into our new Elasticsearch SQL release. Our hope is to allow developers, data scientists, and others that are familiar with the SQL language -- but so far unfamiliar with or unable to use the Elasticsearch query language -- to use the speed, scalability, and full-text power that Elasticsearch offers and others have grown to know and love.
If you’re just getting started using this functionality or the power of Elasticsearch that powers it, here are a few things to try:
- SELECT … ORDER BY SCORE() DESC to be able to sort by the relevance of the search results
- Get all of the full-text magic from tokenization to stemming by using the MATCH operator like SELECT … WHERE MATCH(fieldname, 'some text')
- Connect your favorite JDBC-compatible tool to Elasticsearch with our JDBC driver
- Learn how to use the full power of the Elasticsearch DSL by translating a SQL query you know via the translate API
This is still the very early days of our SQL efforts, so we’re marking it as experimental while we receive feedback from you on any bugs and feature requests. In the future, we hope to release an ODBC driver as well as simplify and build out our existing JDBC driver including other functions and predicates so that we can have better compatibility with other 3rd party tools like BI tools. In the meantime, please do let us know what you’d like to see next and any problems you run into!
Note that this feature is made available in the “default” (non-OSS-only) distribution of Elasticsearch and the REST API -- including the “translate” functionality and the CLI tool are completely free.
Rollups
The Elastic Stack has been increasingly used as a datastore and visualization engine for IoT, performance monitoring, and other metrics data. And while we continue to believe that most data tends to have value over a long period of time, the relative value of the data can decrease as it ages. With the release of 6.3, we can now create a job in Elasticsearch which will periodically roll up aggregate statistics about recently added data and make it available for search.
Rollups are not themselves completely new in the metrics storage world, but we’ve done a few automatic things that we think are pretty neat:
- When you ask Elasticsearch to store a rollup of data like an average, it will automatically store underlying statistics like sum and count so that it can recompute the average on at query time to help you avoid the Simpson’s paradox.
- The fact that we store some of the underlying statistics of the rollup data allows you to query rolled up data and “live” data together in a single query. Oh, and that query uses the same standard DSL you’ve used for years without you having to learn anything new!
Please let us know how this experimental feature works for you!
Note that this feature is made available in the “default” (non-OSS-only) distribution of Elasticsearch and is completely free.
Java 10 Support
With the previous release of Elasticsearch, we announced support for Java 9. Java 9 is a short term release, and has already reached EOL this past March 2018. To continue support of Java short term releases, we are announcing support of Java 10 in Elasticsearch 6.3. Java 10 is also a short term release, and scheduled to reach EOL in September 2018.
As previously mentioned, we strongly advise the majority of users to stick with Java 8, unless they are comfortable with the rapid release cycle of Java short term versions (and EOL dates). Our support matrix is a handy reference for JVM options for Elasticsearch.
Security Updates
This update also fixes two security issues.
Elasticsearch Information Exposure Vulnerability (ESA-2018-10)
In Elasticsearch versions 6.0.0-beta1 to 6.2.4 a disclosure flaw was found in the _snapshot API. When the access_key and security_key parameters are set using the _snapshot API they can be exposed as plain text by users able to query the _snapshot API.
Although it is advised in the 6.X _snapshot API documentation to define the access_key and security_key parameters in the keystore, it is still possible to define them outside of the keystore using the API.Affected Versions: Elasticsearch versions 6.0.0-beta1 to 6.2.4
Elasticsearch Information Exposure Vulnerability (ESA-2018-11)
A sensitive data disclosure flaw was found in the Elasticsearch repository-azure (formerly elasticsearch-cloud-azure) plugin. When the repository-azure plugin is set to log at TRACE level Azure credentials can be inadvertently logged.
Affected Versions: All versions of Elasticsearch
Known issue with Elasticsearch 6.3.x on JDK 10
There appears to be a bug in JDK 10 impacting Elasticsearch 6.3.x (note that the official Docker images are based on JDK 10). The issue appears to impact the C2 compiler on hardware that supports AVX-512 instructions (e.g., Skylake-X CPUs). The issue manifests as a segmentation fault in Java code executing org.apache.util.MergedIterator#pullTop:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fc7ebc050eb, pid=1, tid=151
#
# JRE version: OpenJDK Runtime Environment (10.0.1+10) (build 10.0.1+10)
# Java VM: OpenJDK 64-Bit Server VM (10.0.1+10, mixed mode, tiered, compressed oops, concurrent mark sweep gc, linux-amd64)
# Problematic frame:
# J 14088 c2 org.apache.lucene.util.MergedIterator.pullTop()V (135 bytes) @ 0x00007fc7ebc050eb [0x00007fc7ebc041e0+0x0000000000000f0b]
or executing org.apache.util.MergedIterator#pushTop:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fc7ebc050eb, pid=1, tid=140
#
# JRE version: OpenJDK Runtime Environment (10.0.1+10) (build 10.0.1+10)
# Java VM: OpenJDK 64-Bit Server VM (10.0.1+10, mixed mode, tiered, compressed oops, concurrent mark sweep gc, linux-amd64)
# Problematic frame:
# J 13039 c2 org.apache.lucene.util.MergedIterator.pushTop()V (92 bytes) @ 0x00007f8c43de5371 [0x00007f8c43de4c40+0x0000000000000731]
A workaround is to disable the C2 compiler from using AVX-512 instructions via the JVM option -XX:UseAVX=2. This issue has been taken upstream to the JDK maintainers and the workaround will be necessary until a version of the JDK that is not prone to this issue is released. For additional background, see issue 31425.
Conclusion
Please download Elasticsearch 6.3.0, try it out, and let us know what you think on Twitter (@elastic) or in our forum. You can report any problems on the GitHub issues page.