Snapshotting to Google Cloud Storage (GCS) for Elasticsearch 6.x

edit

Snapshotting to Google Cloud Storage (GCS) for Elasticsearch 6.x

edit

Snapshotting to GCS isn’t currently supported in the Cloud UI, but there is a workaround using Application Default Credentials (https://developers.google.com/identity/protocols/application-default-credentials). This method uses keys built-in to your GCP instances to login to GCS.

To use the Application Default Credentials it is necessary to make a small change to the Instance Template when you create your GCP instances Google Cloud Platform Configuration.

This method will only work for Elasticsearch 6.x clusters. If you are using an earlier Elasticsearch version you will need to upgrade first.

  1. Create an instance template with access to Storage API:

    When creating an Instance Template, under Identity and API access you need to set either: Allow full access to all Cloud APIs or Set access for each API and select Read Write under Storage

    + image::images/ece-gcp-default-auth.png[Creating an instance template with default auth]

  2. Add Snapshot Repository configuration via Advanced Cluster Configuration:

    1. Create an Elasticsearch cluster with the repository-gcs plugin enabled.
    2. Under the Manage tab, click the advanced cluster configuration link.
    3. In the Data textarea, carefully edit the JSON to add a top-level gcp_resources config object with your bucket name for snapshotting (test-ece-auth in this case). Also set enabled under snapshot to true.:
GCP resources cluster data
  1. Press the Data section Save button to store the updated cluster data.
  2. Trigger a plan change by clicking on the Save button under the Plan section.
  3. The cluster should then have snapshots enabled and and will begin snapshotting either immediately (if skip_snapshot is false in the plan) or in 30 minutes (the default snapshot interval).