Elastic Stack 8.11: Introducing a new powerful query language, ES|QL

In 8.11, we are introducing Elasticsearch Query Language (ES|QL), Elastic’s new piped language for data exploration and investigation. ES|QL transforms, enriches, and simplifies your data explorations process. 

Here is what you can expect: 

• Easy start: To begin using ES|QL in Discover, simply select “Try ES|QL” from the data-view picker. It’s user friendly and straightforward. 
• Efficient and easy query building: ES|QL in Discover offers auto-complete and in-app documentation, making it easy to craft powerful queries right from the query bar. 
• Comprehensive and powerful data exploration: Conduct ad-hoc data exploration within Discover. Create aggregations, transform data, enrich data sets, and more, directly from the query builder. Results are presented in a tabular format or as visualizations; it depends on the query you are executing. 
• Contextual visualizations: When writing ES|QL queries in Discover, you’ll receive visual representations powered by the Lens suggestion engine. Your query’s nature determines the type of visualization you get (e.g., a metric, histogram heatmap). 
• Enrichment: Use the enrich command to enhance your query data set with fields from another data set, complete with in-context suggestions for the selected policy (i.e., hinting the matching field and enriched columns).
• In-line visualization editing: Edit ES|QL visualizations directly within Discover and Dashboards. No need to navigate to Lens for quick edits; you can make changes seamlessly. 
• Dashboard integration: Save your ES|QL visualizations to a Dashboard directly from Discover once you’re satisfied with the results. 
• Alerting: Utilize ES|QL for observability and security alerts, setting aggregated values as thresholds. Enhance detection accuracy and receive actionable notifications by emphasizing meaningful trends over isolated incidents, reducing false positives. 

ES|QL in Discover brings efficiency and power to your data investigations, streamlining your path to insights.

A new ES|QL alerting rule type is now available under the existing Elasticsearch rule type. This rule type brings all the new functionalities that are available within the new and powerful language — ES|QL — to Kibana Alerting to allow and unlock new alerting use cases.

With the new type, users will be able to generate a single alert based on defined ES|QL query and preview the query result before saving the rule. When the query returns an empty result, no alerts will be generated.

We are working to introduce a unified inference API that abstracts away the complexity of performing inference on different models that are trained for different tasks. The API introduces a simple, intuitive syntax of the form:

POST /_inference/<task_type>/<model_id>

In 8.11, we are releasing a contained first MVP iteration of this framework. This MVP is in technical preview and initially only supports ELSER. This greatly simplifies the syntax for creating an inference pipeline.

More importantly, in the future the new inference API will support both internal and external models and will integrate with the LLM ecosystem for our users to have the most powerful AI effortlessly and seamlessly at their fingertips, through a unified, self-explanatory API.

You can now edit a Lens visualization without leaving the dashboard instead of navigating back and forth to the Lens editor. A flyout will be open in the dashboard where you can perform any edits to your Lens panels. This new editing experience is more convenient and will save you time since the dashboard will not need to reload when saving your changes.

To manage your Elastic data’s lifecycle, you can use index lifecycle management, which offers a lot of power to fully customize data tier movement, rollover, index settings, downsampling, and more. There are many possibilities and perhaps more customizability than some people need. 

We’ve been working on a built-in simplified and resilient lifecycle implementation for data streams that is available now in 8.11. The main idea with data stream lifecycle is simplicity: We’ve designed this capability from the start to be easy to configure, so we removed as many implementation details as possible from the user’s concern, only exposing configuration that relates to your needs. We’ll only ask you things that are relevant to your use case and your business:

• How long should we keep the data before it gets deleted (retention)? 
• For your Time Series Data Streams, do you want to reduce the granularity of the metrics over time to reduce storage costs (downsample)?

You can set the retention in Kibana’s Index Management page under Data Streams:

Cross-cluster search (CCS) is a great way to unify data across multiple clusters that can be spread all over the world and in different environments. We want it to bring complete search results as often as possible, but maybe some of the remote clusters are having a bad day. We’ll still return partial results whenever we can, but how do you know what’s missing, and which clusters had what errors? 8.11 brings additional search response accounting info, and now we make it easy to find it in Kibana’s Inspector. 

Each visualization that has a search that didn’t return 100% without issues will have a warning that also links directly to the Clusters and shards tab of the Inspector.

We introduced the reroute processor in technical preview status in 8.8, enabling what we call document based routing. If you have documents coming in mixed together, maybe from a Firehose or docker logging driver, you can set up reroute processors to direct each kind of document to a more applicable ingest pipeline so they can be processed correctly for optimal searching. With Elastic 8.11, reroute is now GA. 

If you want to learn more about this capability and see examples, check out this blog by our Observability team.

We have added the option for a document to have multiple vectors within a single field, and to rank by the most similar vector of the vectors within the document. There are many use cases in which this capability is critical, but two stand out as the most popular:

1. Chunking text: Many embedding models limit the size of the text to 512 tokens (often meaning approximately 512 words). This roughly translates to a paragraph length text. Users frequently want to search for the text that contains the most relevant paragraph. To do that, the users create a vector per paragraph and want to rank the documents by the most similar vector within each document, which is now supported.

2. Multiple images: A document frequently contains multiple images (e.g., if the document represents a real estate asset, it will contain multiple images of that asset; if a document represents a product in ecommerce, there will be different images of that product; and if a document represents a person, it will contain multiple images of that person). The user wants to find the most relevant document (e.g., asset, product, or person). The Convolutional Neural Network is designed to generate a vector for each image, and the user wants to rank the docs by the most similar image and vector.

For more information about this exciting and quite unique functionality, see the Adding passage vector search to Lucene blog.

We have increased the number of dimensions supported for dense vector search to 4096 dimensions. As indicated in the past, we do not see an issue with raising this bar even further. The reason we maintain the limit at 4096 dimensions is that we currently don’t see production ready models that require higher dimensionality. If the need rises, we will raise the limit further.

Elasticsearch now supports dense vector search using maximum inner product (MIPS). This option is added to the other supported vector similarity options (Euclidian, Manhattan, Dot Product, and Cosine). Maximum inner product is required for some vector search models, and in particular for some of those used for generative AI and RAG applications that have become a popular use case for using Elasticsearch in recent months.

Elasticsearch now supports a sparse_vector data type, which can be used by the ELSER model. While it is not materially different from the rank_features data type that ELSER has been using so far, the use case is important, popular, and different enough to merit its own data type. If nothing else, it makes it easier to understand that way.

The Exists query returns the documents that have a particular value in a field. The Exists query has been enhanced to include sparse vectors. This is yet another example of how the envelope of services around the actual vector search is just as important in practice as the search itself. We found that users sometimes ingest docs using the ELSER model and then want to rerun only a small portion of the docs that do not have a populated sparse vector. The Exists query would be handy for that.

Users can perform language analysis using the language analyzer plugins that Elasticsearch supports. It is however easier if Elasticsearch already comes with the plugin, and for that reason we have added support for the Persian language stemmer analyzer plugin.

We are taking action to make the use of dense vector search simpler and easier. As part of that we have made dense vector fields indexed by default. It is yet another operation that an administrator needs to do, which we can eliminate, since it is rare that vectors do not need to be indexed. Additionally, we will now dynamically choose the correct number of dimensions based on the first vector indexed. Less configuration, less hassle. Of course, experts can still avail themselves of the full suite of pre-existing options.

Read about these capabilities and more in the release notes.

Existing Elastic Cloud customers can access many of these features directly from the Elastic Cloud console. Not taking advantage of Elastic on cloud? Start a free trial.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.