Elasticsearch and Kibana 8.16: Kibana gets contextual and BBQ speed and savings!

In 8.16, Elastic has continued the pace of innovation in the generative AI space by bringing greater efficiency, scalability, and usability. With BBQ, achieve top-tier query speed and cost savings by reducing memory needs by up to 97%. The newly generally available (GA) inference API offers production-ready access to Elastic’s powerful AI models plus real-time interactions with just a few clicks. And with adaptive resources and flexible chunking strategies, you get peak performance during high demand and zero cost when idle.

Elasticsearch introduces a novel algorithm for quantization of dense vectors, which produces incredible results in terms of query latency, ranking quality, and required computing resources (cost). 

The overall concept is similar to PQ, in the sense that Elasticsearch produces a predictor vector much smaller than the original vector. The initial search is performed with the predictor vector. It then oversamples results for reranking and reranks them using the original vector to produce a subset of results in response to the query. The method in which the predictor vector is produced is completely different from PQ. It is based on generating a bit vector that is much more storage-efficient than the original vector and still has the ability to predict the correct ranking. 

The bottom line is that BBQ produces excellent ranking, achieving >90% recall (compared to brute force on the original vector) with x2 to x5 oversampling. The latency is also superior to other alternatives, including PQ. Perhaps the most important thing is that the RAM needs are between 3% and 5% of those required by using HNSW on the original vectors, reducing costs dramatically. For a more complete description of the algorithm and some test results, check out this blog.

To use it, simply define the index_options to type bbq_hnsw or bbq_flat, and don’t forget to oversample and rerank with the full vector using script_score query. The functionality is still in technical preview, and we tentatively plan to make the experience of using it simpler in the coming releases.

We are thrilled to share that the inference API is now GA. Starting in 8.16, it is now recommended for production, offering production-level stability, robustness, and performance. Elastic’s inference API integrates the state-of-the-art in AI inference while offering unparalleled ease of use. It brings together Elastic’s AI semantic search model — Elastic Learned Sparse EncodeR (ELSER) — as well as your Elastic hosted models and an increasing array of prominent external models and tasks in a unified, lean syntax for use with your Elastic vector database. Used with semantic_text or the vector fields supported by the Elastic vector database, you can perform AI search, reranking, and completion with seamless simplicity. 

In 8.16, we also added streamed completions for improved flows and real-time interactions and GenAI experiences. The inference API offers state-of-the-art AI inference at your fingertips with more services and tasks to follow.

Inference requires computing resources, so it is imperative that your model deployments adapt to the inference load automatically — scale up during high loads, such as during ingest peaks and busy search times, and scale down when the inference load drops. Until now, you had to actively manage this part, even with the Elastic Cloud ML autoscaling on. And how do you control all the parameters involved to achieve the optimal cost/performance tradeoff for your search and ingest needs at any point in time? 

From 8.16, ELSER and the other AI search and natural language processing (NLP) models you use in Elastic automatically adapt resource consumption according to the inference load — providing the performance you need during peak times and reducing the cost during slow periods all the way down to zero cost during idle times. The e5 model is covered under the standard warranty with Elastic.

In addition, no convoluted configurations are required. Through a user-friendly UX, you can provision search-optimized and ingest-optimized model deployments with a one-click selection. An optimized configuration happens transparently behind the scenes. You can also easily plan for more dynamic resources toward search or ingest. We will continue improving this experience. Combined with the flexibility of ML autoscaling on Elastic Cloud and the incredible elasticity of Elastic Cloud Serverless, you are in full control of both performance and cost.

Retrievers is an abstraction that returns top-ranking documents from a query. It was designed to allow nesting and makes it easier to define a query. Retrievers also allow further flexibility in defining the queries. The capability gained popularity since we released it in 8.14, and we’ve gotten requests to allow multiple levels of nesting. For example, users requested combining dense vector (kNN retriever), ELSER, and BM25 (standard retriever) through an RRF retriever and then reranking them with an external service using a text similarity reranker retriever, which is now supported. In 8.16, we also added support for other search functionality like collapse results and highlighting. And we made retrievers, including RRF, generally available.

Elastic 8.16 introduces three powerful, new features designed to make querying faster, easier, and more flexible. Recommended ES|QL queries offer instant guidance with autocomplete options and prebuilt query suggestions that make it perfect for users at any skill level. Sorting by distance now boasts a performance boost of up to 100x for speedy geosearches and top-N queries. And with per-aggregation filtering, you can define unique filters for each aggregation, bringing pinpoint accuracy to your analytics.

Creating queries in the ES|QL editor is now easier than ever. Recommended queries help streamline the process, especially for users unfamiliar with syntax or data structures. This feature reduces query creation time and simplifies the learning curve for both new and experienced users. You can now quickly select recommended queries from the ES|QL help menu or use autocomplete to get started faster.

Kibana 8.16 elevates user experience with powerful new features for Discover, dashboards, and navigation designed to boost productivity and streamline data exploration. Discover now adapts data tables based on data type, making it easier for users to analyze logs. Dashboards gain quick access features with recent views, favorites, and usage insights. Additionally, a new solution-focused left-menu navigation tailors the interface for Search, Observability, and Security users — bringing a more intuitive and efficient experience across Kibana.

Discover in Kibana 8.16 now automatically adjusts data table presentation based on the type of data being explored. For users working with logs, relevant fields like log levels are now automatically displayed in the table or in a custom logs overview in the Discover document viewer. This streamlined, context-aware approach boosts productivity by simplifying data exploration and highlighting key log insights without the need for additional configuration.

Queries using the "event.ingested" field to search data based on its ingestion datetime now enjoy the same performance boost as those using "@timestamp" when using searchable snapshots. This is achieved thanks to an optimization that allows shards to be skipped based on the minimum and maximum values stored in the cluster state for "event.ingested".

Data streams are essential for efficiently ingesting logs and metrics over time, and managing data retention is crucial for maintaining system efficiency and cost-effectiveness. With this release, administrators gain access to two new global settings that allow them to define maximum and default retention periods for all data streams in their deployment, ensuring that outdated data is removed while keeping costs manageable.

Intervals query was designed to replace span queries. It is simpler to define; it provides proximity scores; and in certain cases, it is more performant. Both queries are often used for legal and patent search. In 8.16, we enabled regexp and range in interval queries and increased the maximum clauses of a multiterm intervals query (prefix, wildcard, fuzzy, regexp, and range) from 128 to a configurable parameter (the same one used for span queries).

Group your logs into patterns with log pattern analysis to accelerate time to insights and resolution. Work through thousands of logs and identify the signal in the noise within seconds with Elastic’s artificial intelligence for IT operations (AIOps) capabilities. With the log pattern analysis available from within your dashboards, AIOps are increasingly embedded in your workflows in context. Filtering patterns will adjust the dashboard’s data accordingly, or choose the filtering to transition you in Discover for further exploration.