IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

› › ›

Client Settings

edit

Client Settings

edit

The client that you use to connect to S3 has a number of settings available. The settings have the form s3.client.CLIENT_NAME.SETTING_NAME. By default, s3 repositories use a client named default, but this can be modified using the repository setting client. For example:

PUT _snapshot/my_s3_repository
{
  "type": "s3",
  "settings": {
    "bucket": "my_bucket",
    "client": "my_alternate_client"
  }
}

Most client settings can be added to the elasticsearch.yml configuration file with the exception of the secure settings, which you add to the Elasticsearch keystore. For more information about creating and updating the Elasticsearch keystore, see Secure settings.

For example, if you want to use specific credentials to access S3 then run the following commands to add these credentials to the keystore:

bin/elasticsearch-keystore add s3.client.default.access_key
bin/elasticsearch-keystore add s3.client.default.secret_key
# a session token is optional so the following command may not be needed
bin/elasticsearch-keystore add s3.client.default.session_token

If instead you want to use the instance role or container role to access S3 then you should leave these settings unset. You can switch from using specific credentials back to the default of using the instance role or container role by removing these settings from the keystore as follows:

bin/elasticsearch-keystore remove s3.client.default.access_key
bin/elasticsearch-keystore remove s3.client.default.secret_key
# a session token is optional so the following command may not be needed
bin/elasticsearch-keystore remove s3.client.default.session_token

All client secure settings of this plugin are reloadable. After you reload the settings, the internal s3 clients, used to transfer the snapshot contents, will utilize the latest settings from the keystore. Any existing s3 repositories, as well as any newly created ones, will pick up the new values stored in the keystore.

In-progress snapshot/restore tasks will not be preempted by a reload of the client’s secure settings. The task will complete using the client as it was built when the operation started.

The following list contains the available client settings. Those that must be stored in the keystore are marked as "secure" and are reloadable; the other settings belong in the elasticsearch.yml file.

access_key (Secure, reloadable): An S3 access key. If set, the secret_key setting must also be specified. If unset, the client will use the instance or container role instead.
secret_key (Secure, reloadable): An S3 secret key. If set, the access_key setting must also be specified.
session_token (Secure, reloadable): An S3 session token. If set, the access_key and secret_key settings must also be specified.
endpoint: The S3 service endpoint to connect to. This defaults to s3.amazonaws.com but the AWS documentation lists alternative S3 endpoints. If you are using an S3-compatible service then you should set this to the service’s endpoint.
protocol: The protocol to use to connect to S3. Valid values are either http or https. Defaults to https.
proxy.host: The host name of a proxy to connect to S3 through.
proxy.port: The port of a proxy to connect to S3 through.
proxy.username (Secure, reloadable): The username to connect to the proxy.host with.
proxy.password (Secure, reloadable): The password to connect to the proxy.host with.
read_timeout: The socket timeout for connecting to S3. The value should specify the unit. For example, a value of 5s specifies a 5 second timeout. The default value is 50 seconds.
max_retries: The number of retries to use when an S3 request fails. The default value is 3.
use_throttle_retries: Whether retries should be throttled (i.e. should back off). Must be true or false. Defaults to true.
path_style_access: Whether to force the use of the path style access pattern. If true, the path style access pattern will be used. If false, the access pattern will be automatically determined by the AWS Java SDK (See AWS documentation for details). Defaults to false.

In versions 7.0, 7.1, 7.2 and 7.3 all bucket operations used the now-deprecated path style access pattern. If your deployment requires the path style access pattern then you should set this setting to true when upgrading.

disable_chunked_encoding: Whether chunked encoding should be disabled or not. If false, chunked encoding is enabled and will be used where appropriate. If true, chunked encoding is disabled and will not be used, which may mean that snapshot operations consume more resources and take longer to complete. It should only be set to true if you are using a storage service that does not support chunked encoding. See the AWS Java SDK documentation for details. Defaults to false.

S3-compatible services

edit

There are a number of storage systems that provide an S3-compatible API, and the repository-s3 plugin allows you to use these systems in place of AWS S3. To do so, you should set the s3.client.CLIENT_NAME.endpoint setting to the system’s endpoint. This setting accepts IP addresses and hostnames and may include a port. For example, the endpoint may be 172.17.0.2 or 172.17.0.2:9000. You may also need to set s3.client.CLIENT_NAME.protocol to http if the endpoint does not support HTTPS.

Minio is an example of a storage system that provides an S3-compatible API. The repository-s3 plugin allows Elasticsearch to work with Minio-backed repositories as well as repositories stored on AWS S3. Other S3-compatible storage systems may also work with Elasticsearch, but these are not tested or supported.

« Getting Started Repository Settings »