Automated Snapshots in Elastic Cloud Enterprise

By
Igor Kupczyński

Elastic Cloud Enterprise supports automated snapshots to your own repository. This is a convenient way to back up your data and to make sure that any changes to your clusters are easily reversible. 

Introduction

Elasticsearch stores data, so it is important to back it up as a protection against catastrophic failures. You can configure your indices to use replicas so you can withstand a node or server failure, but this is not enough in the case of a datacenter-wide issue or an operator error. To make the back up process simple Elasticsearch offers the snapshot API. A snapshot represents a point-in-time version of an index or even of all the indices in the cluster, including internal ones like .security. Snapshots are incremental — that is, they store a delta between the latest snapshot and the current view of the index — to ensure that backups can be done in a fast and efficient manner. Snapshots can then be restored to a new, empty cluster or even to the same cluster using the restore API. Check the Elasticsearch documentation to read more about these APIs. 

Clusters created with Elastic Cloud Enterprise have full support for the Elasticsearch snapshot and restore APIs. You can add repositories, take snapshots, and restore snapshots according to your needs. However, the process would be tedious when managing dozens or hundreds of clusters without support for automated snapshots. 

Automated snapshots in Elastic Cloud Enterprise

In Elastic Cloud Enterprise we want to make the process of managing multiple clusters as easy as possible. Therefore, you can create one or more shared, system-wide snapshot repositories and then link clusters created with ECE to them. 

Using an ECE managed snapshot repository provides a couple of benefits over the classic manually managed repositories. Firstly, clusters linked to a shared repository will automatically take their snapshots every half an hour. Since the snapshots are incremental this process is usually very quick and doesn't affect the cluster’s performance in any significant way. 

Moreover, Elastic Cloud Enterprise will take a snapshot before any changes are made to the cluster topology — e.g. a script settings change, up/down scale or addition of new nodes. This means that in the case of an unintended or destructive change a cluster can easily be restored to the previous state. 

Elastic Cloud Enterprise will keep the 100 most recent snapshots for each cluster, which gives you the possibility to restore the state of the cluster for the next two days. Additionally, in the event that partial snapshots are present, the most recent full snapshot is always maintained. 

Elasticsearch itself supports multiple types of repositories, such as AWS S3, Google Cloud Storage or even shared file systems. As of this article, Elastic Cloud Enterprise supports only automated snapshots to S3-compatible storage. You can also use custom AWS S3 repositories by adding them to your Elasticsearch cluster directly via the snapshot and restore API. Support for additional repositories, such as Google Cloud Storage and Microsoft Azure Storage, is planned for a future release.

Step by step guide

1. Create the shared repository 

First lets configure the shared repository. 

  • Go to Platform →  Repositories and select Create Repository 
    ece-1-create-snapshot-repository.png
  • You can either use a dedicated S3 configuration screen or select Repository Type: Advanced and provide all the details in json. This json should follow the regular Elasticsearch S3 repository format described in the documentation. The Advanced screen gives you an opportunity to configure, among other settings,  the S3 endpoint. This can be useful if you have your own S3-compatible storage. 
    ece-2-create-snapshot-repository.png
  • Finally click Save and confirm that the repository was added. 
    ece-3-snapshot-repository-configuration.png

In this example we've added an S3 repository named blogpost. Now we can reference it from the clusters. 

2. Link the cluster to it 

During new cluster creation one can select a shared repository to use. 

ece-4-select-snapshot-repository.png

Similarly, if you have existing clusters — created before the shared repository was added — you can link them as well. Go to a cluster, select Manage and then select a Snapshot repository and click Save repository. 

ece-5-select-snapshot-repository.png

3. Automated Snapshots 

When the repository is linked, then all the plan changes, e.g. scaling the cluster or changing settings will include the additional perform-snapshot step. 

ece-6-perform-shanpshot-as-part-of-plan.png

This way you can safely change your clusters, knowing that there is a point-in-time snapshot just before the plan change. 

Additionally, an automated snapshot is performed every 30 minutes and the last 100 snapshots are kept — which gives you around 48 hours to restore from backup if anything goes wrong. 

You can monitor the snapshot status with the Snapshots tab in the cluster details. 

ece-7-list-snapshots.png

ece-8-list-snapshots.png

Summary

To sum up, the shared repositories feature of ECE makes managing a set of Elasticsearch clusters easier and safer. Elastic Cloud Enterprise will take a snapshot before any changes are made to the cluster topology. Moreover, an incremental snapshot is taken every half an hour. Unintended or destructive changes can be easily reverted, it is easy to restore clusters to their previous states.

Learn more

Check Elastic Cloud Enterprise documentation https://www.elastic.co/guide/en/cloud-enterprise/current/index.html.