Elastic Stack 8.9: Faster cross-cluster searches and aggregations on metrics

As part of the time series database (TSDB) project, relating to optimizing Elasticsearch for metrics data, we set to improve the latency of aggregations that are frequently used with metrics data. We focused on Kubernetes metrics and achieved improvements of between one to two orders of magnitude in aggregation latency (i.e., running the relevant aggregations takes between 10% to 1% of the time it took prior to 8.9).

Let’s examine the details and the relevant public nightly benchmarks:

• We improved the default refresh behavior of Elasticsearch. By default, a refresh only happens if a shard is search active, and a shard goes search-idle after 30 seconds of search activity. When a shard is search-idle, no scheduled refresh happens, and this favors indexing throughput. However, searching a search-idle shard can increase query time significantly — this is noticeable when loading visualizations in Kibana because after some time of inactivity, dashboards load slowly. We made two changes that reduced latency of searches with aggregations by over 90%, because shards were search-idle.
  • We improved the can match phase of the search api to sometimes determine whether a query doesn’t match with a shard without doing any search related i/o. For specific queries that query a field that doesn’t exist or a constant keyword field that doesn’t match, we can skip entire shards without doing any search related i/o on data nodes. This improves the default refresh behavior because shards remain search-idle and no scheduled refresh happens.
  • When a shard is search-idle and a refresh does need to happen, the refresh is executed immediately, instead of having the refresh scheduler to kick in. This reduces latency when searching idle shards and reduces query time by over 1 second (the exact improvement depends on the number of shards allocated on a data node).

We’ve also improved remote searching against Searchable Snapshots (Cold or Frozen tier data) with an optimization to do shard skipping. Local searches (non CCS) could filter out (skip) irrelevant shards using cluster state information, comparing the time range of the search request to min and max timestamps in shard metadata. CCS did not support this shard skipping mechanism when not minimizing round trips, until now. 

In 8.9, we execute the can_match phase of search in the remote clusters. This makes search behave similarly with and without CCS, in that the coordinating node can determine which shards to skip, which is particularly important for cold and frozen indices. This can improve situations where there is latency between the local and remote clusters by not hitting remote clusters as frequently. Note this optimization is relevant only when not minimizing round trips. Cross-Cluster Searches with minimize round trips already had this capability. In 8.9, you’re covered either way.

Index Lifecycle Management (ILM) automates your data management chores, performing rollover, shrink, merge, downsample operations, and more, and moving your data from one tier to the next so you can optimize for cost and performance tradeoffs. It does this through a series of actions (rollover, shrink, etc.) and steps (wait-for-active-shards, etc.). If ILM gets stuck in a step for too long, data can pile up in the wrong tier and lead to problems. 

To help you notice earlier that ILM is stuck, we’ve enhanced the ILM Health Indicator in the Health Report API. New rules will check all the indices managed by an ILM policy according to the current action and step where they are at any time. For instance, if an index has been on the rollover > wait-for-active-shards combo for more than 24 hours or the step wait-for-active-shards has been retried more than 100 times, the indicator will turn yellow.

The time and retries thresholds can be changed using the health indicator settings. 

This indicator will also show up in the Elastic Cloud Deployments console. You can more quickly identify an issue that could be causing cluster performance or stability issues. As with the rest of the Health report, we’ll indicate the possible impact and include a link to a relevant troubleshooting guide.

A popular use case for geographical data is that an asset generates a time series of positions, each with its own timestamp, and the system allows tracking the asset based on that time series. Elasticsearch now supports a simplification algorithm, to allow generating a line based on a smaller subset of position points (defined by the ‘size’ parameter), while minimizing the resulting inaccuracy. This capability is based on the time_series index mode infrastructure (TSDS). The simplification reduces memory consumption and supports creating complex geo_lines based on large sets of positions. Geo line simplification is released in tech preview.

Elastic 8.9 delivers a number of enhancements to existing Kibana features — we are highlighting two of them below. Further, “Rules as code” significantly simplifies managing rules across multiple nodes in your cluster by leveraging Terraform to replicate them and apply version control — this facilitates maintenance of your monitoring rules for observability and security issues. 

Kibana stacked bar chart legend
The order of stacked bar chart legends is now based on popularity with the most important element displayed on top. You no longer need to scroll to the bottom of the legend to find the most popular data in the chart.

Previously:

Existing Elastic Cloud customers can access many of these features directly from the Elastic Cloud console. Not taking advantage of Elastic on cloud? Start a free trial.