IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Restore from snapshot Addressing repeated snapshot policy failures »

› ›

Multiple deployments writing to the same snapshot repository

edit

Multiple deployments writing to the same snapshot repository

edit

Multiple Elasticsearch deployments are writing to the same snapshot repository. Elasticsearch doesn’t support this configuration and only one cluster is allowed to write to the same repository. See Repository contents for potential side-effects of corruption of the repository contents, which may not be resolved by the following guide. To remedy the situation mark the repository as read-only or remove it from all the other deployments, and re-add (recreate) the repository in the current deployment:

Fixing the corrupted repository will entail making changes in multiple deployments that write to the same snapshot repository. Only one deployment must be writing to a repository. The deployment that will keep writing to the repository will be called the "primary" deployment (the current cluster), and the other one(s) where we’ll mark the repository read-only as the "secondary" deployments.

First mark the repository as read-only on the secondary deployments:

Use Kibana

Log in to the Elastic Cloud console.
On the Elasticsearch Service panel, click the name of your deployment.

If the name of your deployment is disabled your Kibana instances might be unhealthy, in which case please contact Elastic Support. If your deployment doesn’t include Kibana, all you need to do is enable it first.
Open your deployment’s side navigation menu (placed under the Elastic logo in the upper left corner) and go to Stack Management > Snapshot and Restore > Repositories.
The repositories table should now be visible. Click on the pencil icon at the right side of the repository to be marked as read-only. On the Edit page that opened scroll down and check "Read-only repository". Click "Save". Alternatively if deleting the repository altogether is preferable, select the checkbox at the left of the repository name in the repositories table and click the "Remove repository" red button at the top left of the table.

At this point, it’s only the primary (current) deployment that has the repository marked as writeable. Elasticsearch sees it as corrupt, so the repository needs to be removed and added back so that Elasticsearch can resume using it:

Note that we’re now configuring the primary (current) deployment.

Open the primary deployment’s side navigation menu (placed under the Elastic logo in the upper left corner) and go to Stack Management > Snapshot and Restore > Repositories.
Click on the pencil icon at the right side of the repository. On the Edit page that opened scroll down and click "Save", without making any changes to the existing settings.

Fixing the corrupted repository will entail making changes in multiple clusters that write to the same snapshot repository. Only one cluster must be writing to a repository. Let’s call the cluster we want to keep writing to the repository the "primary" cluster (the current cluster), and the other one(s) where we’ll mark the repository as read-only the "secondary" clusters.

Let’s first work on the secondary clusters:

Get the configuration of the repository:

GET _snapshot/my-repo

The reponse will look like this:

{
  "my-repo": { 
    "type": "s3",
    "settings": {
      "bucket": "repo-bucket",
      "client": "elastic-internal-71bcd3",
      "base_path": "myrepo"
    }
  }
}

Represents the current configuration for the repository.

Using the settings retrieved above, add the readonly: true option to mark it as read-only:

PUT _snapshot/my-repo
{
    "type": "s3",
    "settings": {
      "bucket": "repo-bucket",
      "client": "elastic-internal-71bcd3",
      "base_path": "myrepo",
      "readonly": true 
    }
}

Marks the repository as read-only.

Alternatively, deleting the repository is an option using:

response = client.snapshot.delete_repository(
  repository: 'my-repo'
)
puts response

DELETE _snapshot/my-repo

The response will look like this:

{
  "acknowledged": true
}

At this point, it’s only the primary (current) cluster that has the repository marked as writeable. Elasticsearch sees it as corrupt though so let’s recreate it so that Elasticsearch can resume using it. Note that now we’re configuring the primary (current) cluster:

Get the configuration of the repository and save its configuration as we’ll use it to recreate the repository:
```
GET _snapshot/my-repo
```

Using the configuration we obtained above, let’s recreate the repository:

PUT _snapshot/my-repo
{
  "type": "s3",
  "settings": {
    "bucket": "repo-bucket",
    "client": "elastic-internal-71bcd3",
    "base_path": "myrepo"
  }
}

The response will look like this:

{
  "acknowledged": true
}

« Restore from snapshot Addressing repeated snapshot policy failures »