Elastic Cloud Enterprise 2.5: Dedicated coordinating layer, snapshot lifecycle management, and more

We’re excited to announce the release of Elastic Cloud Enterprise (ECE) 2.5! This release improves the experience of managing your deployments with a dedicated coordinating layer, support for snapshot lifecycle management (SLM), and more.

Better performance and scalability with dedicated coordinating nodes

There are many reasons to consider having a dedicated, independently scalable set of nodes to handle ingest pipelines and coordinate search queries. As we continue to enhance Elasticsearch ingest node capabilities to support features such as enrich and circle processors, you’ll likely want to have dedicated nodes to execute the ingest pipelines. If you have a search-intense use case or are using bulk indexing, you’ll also likely want to have dedicated nodes to coordinate those requests to free the data nodes from handling these tasks.

ECE already allowed you to configure a dedicated ingest layer, and for this release we improved this functionality to handle both ingest pipelines and coordination of search requests. You can now include and enable a dedicated coordinating layer in your deployment templates. This layer is also included in the preconfigured deployment templates, and you can scale it according to your use case without having to scale your data nodes. To deploy nodes on hardware best suited for your operations, you can use allocator tags and filters, which will optimize resource utilization and improve performance.

In order to ensure your data is still accessible in case there’s a problem with the coordination layer — such as the coordinating nodes not being available — ECE will automatically fall back and use the data node to coordinate the search request. Ingest pipelines, on the other hand, will not be executed. And while you can continue to search data, your writes can be affected, so it’s important to configure the coordinating layer to be highly available and properly sized.

Support for Elastic Stack snapshot lifecycle management

In Elastic Stack 7.4, we introduced support for SLM, which allows you to manage snapshot repositories. Since then, we’ve introduced additional improvements to SLM policy configuration, allowing you to automate snapshot lifecycles as well as the restore process. Starting from ECE 2.5 and Elastic Stack version 7.6, you can leverage all the great SLM features in ECE-hosted deployments.

Additionally, ECE will officially cut over from the existing snapshot mechanism (also known as “the snapshotter”) to use SLM in new deployments using Elastic Stack versions 7.6 or above and migrate existing deployments when they’re upgraded to a compatible version. 

ECE has the same easy-to-use interface as before but with an additional option to use cron expression to configure when a snapshot should be taken. Plus, it offers a deep link to the SLM interface in Kibana with a more robust mechanism to manage your own custom snapshot repositories and SLM policies.

This will allow you to enjoy additional capabilities that will be introduced to the SLM feature, such as tight integration with index lifecycle management (ILM).

Migrating from index curation to ILM

With ECE 2.0, we introduced support for hot-warm deployment templates, which are ideal for various time series use cases such as security, observability, and many more. In those templates we introduced a new index curation feature that automatically reallocates indices from hot to warm nodes based on the index pattern after a certain configured period. Since then we’ve released the ILM feature, which offers a much more robust mechanism to manage your indices through different phases.

When creating a new deployment with compatible Elastic Stack versions, the default behavior is to use ILM. When upgrading a deployment to a compatible version, we previously provided documentation describing how to migrate from index curation to ILM. That migration is a multistep procedure that requires changes to the advanced configuration section and can be cumbersome. With this version, we've introduced a wizard that allows you to make a smooth transition with just a few clicks. 

When you’re ready, you can initiate the migration from the deployment page, set the name of the ILM policies you wish to be created, edit the period if needed — and that’s it! We’ll automatically create ILM policies to match your index curation configuration and associate each policy to the relevant index pattern. The behavior will be exactly the same, and you don’t need to make any additional changes. From that point on, you can further customize the newly created ILM policies in Kibana and take advantage of all the great features ILM has to offer. 

We encourage you to migrate to ILM since we have deprecated support for index curation and will remove it in the next ECE major version, so take the time to do so as soon as possible.

Custom Kibana plugins

If you’d like to leverage Kibana’s extension points to use a custom visualization or build a new Kibana app, you can write your own custom plugin or use one of the existing known plugins.

As part of this release, we published a step-by-step guide about how to add your custom Kibana plugins in ECE stack packs, giving you control over which plugins are installed. You'll need to verify that the plugin is compatible with the stack pack version. Then, you can add the plugin to the stack pack, repackage the stack pack, and upload it to make it available to your users and teams. The plugin will be installed automatically in deployments created using that stack pack.

This functionality is supported starting from ECE 2.0 with the Elastic Stack pack in versions 6.8 and above.

Try it out

There are plenty of additional improvements and important bug fixes you can read about in the release notes. And if you’re not using ECE, you can take it for a spin with a free 30-day trial!