Create follower API

Creates a cross-cluster replication follower index.

Request

resp = client.ccr.follow(
    index="<follower_index>",
    wait_for_active_shards="1",
    remote_cluster="<remote_cluster>",
    leader_index="<leader_index>",
)
print(resp)

const response = await client.ccr.follow({
  index: "<follower_index>",
  wait_for_active_shards: 1,
  remote_cluster: "<remote_cluster>",
  leader_index: "<leader_index>",
});
console.log(response);

PUT /<follower_index>/_ccr/follow?wait_for_active_shards=1
{
  "remote_cluster" : "<remote_cluster>",
  "leader_index" : "<leader_index>"
}

Prerequisites

edit

If the Elasticsearch security features are enabled, you must have write, monitor, and manage_follow_index index privileges for the follower index. You must have read and monitor index privileges for the leader index. You must also have manage_ccr cluster privileges on the cluster that contains the follower index. For more information, see Security privileges.

Description

edit

This API creates a new follower index that is configured to follow the referenced leader index. When this API returns, the follower index exists, and cross-cluster replication starts replicating operations from the leader index to the follower index.

Path parameters

edit

<follower_index>: (Required, string) The name of the follower index.

Query parameters

edit

wait_for_active_shards: (Optional, integer) Specifies the number of shards to wait on being active before responding. This defaults to waiting on none of the shards to be active. A shard must be restored from the leader index before being active. Restoring a follower shard requires transferring all the remote Lucene segment files to the follower index.
master_timeout: (Optional, time units) Period to wait for the master node. If the master node is not available before the timeout expires, the request fails and returns an error. Defaults to 30s. Can also be set to -1 to indicate that the request should never timeout.

Request body

edit

leader_index: (Required, string) The name of the index in the leader cluster to follow.
remote_cluster: (Required, string) The remote cluster containing the leader index.
data_stream_name: (Optional, string) If the leader index is part of a data stream, the name to which the local data stream for the followed index should be renamed. For example, A request like:

resp = client.ccr.follow(
    index=".ds-logs-mysql-default_copy-2022-01-01-000001",
    remote_cluster="remote_cluster",
    leader_index=".ds-logs-mysql-default-2022-01-01-000001",
    data_stream_name="logs-mysql-default_copy",
)
print(resp)

const response = await client.ccr.follow({
  index: ".ds-logs-mysql-default_copy-2022-01-01-000001",
  remote_cluster: "remote_cluster",
  leader_index: ".ds-logs-mysql-default-2022-01-01-000001",
  data_stream_name: "logs-mysql-default_copy",
});
console.log(response);

PUT /.ds-logs-mysql-default_copy-2022-01-01-000001/_ccr/follow
{
  "remote_cluster" : "remote_cluster",
  "leader_index" : ".ds-logs-mysql-default-2022-01-01-000001",
  "data_stream_name": "logs-mysql-default_copy"
}

Replicates the leader index .ds-logs-mysql-default-2022-01-01-000001 into the follower index .ds-logs-mysql-default_copy-2022-01-01-000001 and will do so using the data stream logs-mysql-default_copy, as opposed to the original leader data stream name of logs-mysql-default.

settings: (object) Settings to override from the leader index. Note that certain settings can not be overrode (e.g., index.number_of_shards).

max_read_request_operation_count: (integer) The maximum number of operations to pull per read from the remote cluster.
max_outstanding_read_requests: (long) The maximum number of outstanding reads requests from the remote cluster.
max_read_request_size: (byte value) The maximum size in bytes of per read of a batch of operations pulled from the remote cluster.
max_write_request_operation_count: (integer) The maximum number of operations per bulk write request executed on the follower.
max_write_request_size: (byte value) The maximum total bytes of operations per bulk write request executed on the follower.
max_outstanding_write_requests: (integer) The maximum number of outstanding write requests on the follower.
max_write_buffer_count: (integer) The maximum number of operations that can be queued for writing. When this limit is reached, reads from the remote cluster will be deferred until the number of queued operations goes below the limit.
max_write_buffer_size: (byte value) The maximum total bytes of operations that can be queued for writing. When this limit is reached, reads from the remote cluster will be deferred until the total bytes of queued operations goes below the limit.
max_retry_delay: (time value) The maximum time to wait before retrying an operation that failed exceptionally. An exponential backoff strategy is employed when retrying.
read_poll_timeout: (time value) The maximum time to wait for new operations on the remote cluster when the follower index is synchronized with the leader index. When the timeout has elapsed, the poll for operations will return to the follower so that it can update some statistics. Then the follower will immediately attempt to read from the leader again.

Default values

edit

The following output from the follow info api describes all the default values for the above described index follow request parameters:

{
  "follower_indices" : [
    {
      "parameters" : {
        "max_read_request_operation_count" : 5120,
        "max_read_request_size" : "32mb",
        "max_outstanding_read_requests" : 12,
        "max_write_request_operation_count" : 5120,
        "max_write_request_size" : "9223372036854775807b",
        "max_outstanding_write_requests" : 9,
        "max_write_buffer_count" : 2147483647,
        "max_write_buffer_size" : "512mb",
        "max_retry_delay" : "500ms",
        "read_poll_timeout" : "1m"
      }
    }
  ]
}

Examples

edit

This example creates a follower index named follower_index:

resp = client.ccr.follow(
    index="follower_index",
    wait_for_active_shards="1",
    remote_cluster="remote_cluster",
    leader_index="leader_index",
    settings={
        "index.number_of_replicas": 0
    },
    max_read_request_operation_count=1024,
    max_outstanding_read_requests=16,
    max_read_request_size="1024k",
    max_write_request_operation_count=32768,
    max_write_request_size="16k",
    max_outstanding_write_requests=8,
    max_write_buffer_count=512,
    max_write_buffer_size="512k",
    max_retry_delay="10s",
    read_poll_timeout="30s",
)
print(resp)

const response = await client.ccr.follow({
  index: "follower_index",
  wait_for_active_shards: 1,
  remote_cluster: "remote_cluster",
  leader_index: "leader_index",
  settings: {
    "index.number_of_replicas": 0,
  },
  max_read_request_operation_count: 1024,
  max_outstanding_read_requests: 16,
  max_read_request_size: "1024k",
  max_write_request_operation_count: 32768,
  max_write_request_size: "16k",
  max_outstanding_write_requests: 8,
  max_write_buffer_count: 512,
  max_write_buffer_size: "512k",
  max_retry_delay: "10s",
  read_poll_timeout: "30s",
});
console.log(response);

PUT /follower_index/_ccr/follow?wait_for_active_shards=1
{
  "remote_cluster" : "remote_cluster",
  "leader_index" : "leader_index",
  "settings": {
    "index.number_of_replicas": 0
  },
  "max_read_request_operation_count" : 1024,
  "max_outstanding_read_requests" : 16,
  "max_read_request_size" : "1024k",
  "max_write_request_operation_count" : 32768,
  "max_write_request_size" : "16k",
  "max_outstanding_write_requests" : 8,
  "max_write_buffer_count" : 512,
  "max_write_buffer_size" : "512k",
  "max_retry_delay" : "10s",
  "read_poll_timeout" : "30s"
}

The API returns the following result:

{
  "follow_index_created" : true,
  "follow_index_shards_acked" : true,
  "index_following_started" : true
}

« Get cross-cluster replication stats API Pause follower API »