Google Cloud Storage repository

The Google Cloud Storage service uses the concept of a bucket as a container for all the data. Buckets are usually created using the Google Cloud Platform Console. This repository type does not automatically create buckets.

To create a new bucket:

For more detailed instructions, see the Google Cloud documentation.

The repository must authenticate the requests it makes to the Google Cloud Storage service. It is common for Google client libraries to employ a strategy named application default credentials. However, that strategy is only partially supported by Elasticsearch. The repository operates under the Elasticsearch process, which runs with the security manager enabled. The security manager obstructs the "automatic" credential discovery when the environment variable GOOGLE_APPLICATION_CREDENTIALS is used to point to a local file on disk. It can, however, retrieve the service account that is attached to the resource that is running Elasticsearch, or fall back to the default service account that Compute Engine, Kubernetes Engine or App Engine provide. Alternatively, you must configure service account credentials if you are using an environment that does not support automatic credential discovery.

You have to obtain and provide service account credentials manually.

For detailed information about generating JSON service account files, see the Google Cloud documentation. Note that the PKCS12 format is not supported by this repository type.

Here is a summary of the steps:

A JSON service account file looks like this:

{
  "type": "service_account",
  "project_id": "your-project-id",
  "private_key_id": "...",
  "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
  "client_email": "service-account-for-your-repository@your-project-id.iam.gserviceaccount.com",
  "client_id": "...",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/your-bucket@your-project-id.iam.gserviceaccount.com"
}

To provide this file to the repository, it must be stored in the Elasticsearch keystore. You must add a file setting with the name gcs.client.NAME.credentials_file using the add-file subcommand. NAME is the name of the client configuration for the repository. The implicit client name is default, but a different client name can be specified in the repository settings with the client key.

For example, if you added a gcs.client.my_alternate_client.credentials_file setting in the keystore, you can configure a repository to use those credentials like this:

resp = client.snapshot.create_repository(
    name="my_gcs_repository",
    repository={
        "type": "gcs",
        "settings": {
            "bucket": "my_bucket",
            "client": "my_alternate_client"
        }
    },
)
print(resp)

const response = await client.snapshot.createRepository({
  name: "my_gcs_repository",
  repository: {
    type: "gcs",
    settings: {
      bucket: "my_bucket",
      client: "my_alternate_client",
    },
  },
});
console.log(response);

PUT _snapshot/my_gcs_repository
{
  "type": "gcs",
  "settings": {
    "bucket": "my_bucket",
    "client": "my_alternate_client"
  }
}

The credentials_file settings are reloadable. You can define these settings before the node is started, or call the Nodes reload secure settings API after the settings are defined to apply them to a running node.

After you reload the settings, the internal gcs clients, which are used to transfer the snapshot contents, utilize the latest settings from the keystore.

The service account used to access the bucket must have the "Writer" access to the bucket: