Create a snapshot

This guide shows you how to take snapshots of a running cluster. You can later restore a snapshot to recover or transfer its data.

In this guide, you’ll learn how to:

The guide also provides tips for creating dedicated cluster state snapshots and taking snapshots at different time intervals.

Snapshot lifecycle management (SLM) is the easiest way to regularly back up a cluster. An SLM policy automatically takes snapshots on a preset schedule. The policy can also delete snapshots based on retention rules you define.

The following cluster privileges control access to the SLM actions when Elasticsearch security features are enabled:

You can create and manage roles to assign these privileges through Kibana Management.

To grant the privileges necessary to create and manage SLM policies and snapshots, you can set up a role with the manage_slm and cluster:admin/snapshot/* cluster privileges and full access to the SLM history indices.

For example, the following request creates an slm-admin role:

resp = client.security.put_role(
    name="slm-admin",
    cluster=[
        "manage_slm",
        "cluster:admin/snapshot/*"
    ],
    indices=[
        {
            "names": [
                ".slm-history-*"
            ],
            "privileges": [
                "all"
            ]
        }
    ],
)
print(resp)

const response = await client.security.putRole({
  name: "slm-admin",
  cluster: ["manage_slm", "cluster:admin/snapshot/*"],
  indices: [
    {
      names: [".slm-history-*"],
      privileges: ["all"],
    },
  ],
});
console.log(response);

POST _security/role/slm-admin
{
  "cluster": [ "manage_slm", "cluster:admin/snapshot/*" ],
  "indices": [
    {
      "names": [ ".slm-history-*" ],
      "privileges": [ "all" ]
    }
  ]
}

To grant read-only access to SLM policies and the snapshot history, you can set up a role with the read_slm cluster privilege and read access to the snapshot lifecycle management history indices.

For example, the following request creates a slm-read-only role:

resp = client.security.put_role(
    name="slm-read-only",
    cluster=[
        "read_slm"
    ],
    indices=[
        {
            "names": [
                ".slm-history-*"
            ],
            "privileges": [
                "read"
            ]
        }
    ],
)
print(resp)

const response = await client.security.putRole({
  name: "slm-read-only",
  cluster: ["read_slm"],
  indices: [
    {
      names: [".slm-history-*"],
      privileges: ["read"],
    },
  ],
});
console.log(response);

POST _security/role/slm-read-only
{
  "cluster": [ "read_slm" ],
  "indices": [
    {
      "names": [ ".slm-history-*" ],
      "privileges": [ "read" ]
    }
  ]
}

To manage SLM in Kibana, go to the main menu and click Stack Management > Snapshot and Restore > Policies. To create a policy, click Create policy.

You can also manage SLM using the SLM APIs. To create a policy, use the create SLM policy API.

The following request creates a policy that backs up the cluster state, all data streams, and all indices daily at 1:30 a.m. UTC.

resp = client.slm.put_lifecycle(
    policy_id="nightly-snapshots",
    schedule="0 30 1 * * ?",
    name="<nightly-snap-{now/d}>",
    repository="my_repository",
    config={
        "indices": "*",
        "include_global_state": True
    },
    retention={
        "expire_after": "30d",
        "min_count": 5,
        "max_count": 50
    },
)
print(resp)

const response = await client.slm.putLifecycle({
  policy_id: "nightly-snapshots",
  schedule: "0 30 1 * * ?",
  name: "<nightly-snap-{now/d}>",
  repository: "my_repository",
  config: {
    indices: "*",
    include_global_state: true,
  },
  retention: {
    expire_after: "30d",
    min_count: 5,
    max_count: 50,
  },
});
console.log(response);

PUT _slm/policy/nightly-snapshots
{
  "schedule": "0 30 1 * * ?",       
  "name": "<nightly-snap-{now/d}>", 
  "repository": "my_repository",    
  "config": {
    "indices": "*",                 
    "include_global_state": true    
  },
  "retention": {                    
    "expire_after": "30d",
    "min_count": 5,
    "max_count": 50
  }
}

You can manually run an SLM policy to immediately create a snapshot. This is useful for testing a new policy or taking a snapshot before an upgrade. Manually running a policy doesn’t affect its snapshot schedule.

To run a policy in Kibana, go to the Policies page and click the run icon under the Actions column. You can also use the execute SLM policy API.

POST _slm/policy/nightly-snapshots/_execute

The snapshot process runs in the background. To monitor its progress, see Monitor a snapshot.

SLM snapshot retention is a cluster-level task that runs separately from a policy’s snapshot schedule. To control when the SLM retention task runs, configure the slm.retention_schedule cluster setting.

resp = client.cluster.put_settings(
    persistent={
        "slm.retention_schedule": "0 30 1 * * ?"
    },
)
print(resp)

response = client.cluster.put_settings(
  body: {
    persistent: {
      'slm.retention_schedule' => '0 30 1 * * ?'
    }
  }
)
puts response

const response = await client.cluster.putSettings({
  persistent: {
    "slm.retention_schedule": "0 30 1 * * ?",
  },
});
console.log(response);

PUT _cluster/settings
{
  "persistent" : {
    "slm.retention_schedule" : "0 30 1 * * ?"
  }
}

To immediately run the retention task, use the execute SLM retention policy API.

resp = client.slm.execute_retention()
print(resp)

response = client.slm.execute_retention
puts response

const response = await client.slm.executeRetention();
console.log(response);

POST _slm/_execute_retention

An SLM policy’s retention rules only apply to snapshots created using the policy. Other snapshots don’t count toward the policy’s retention limits.

We recommend you include retention rules in your SLM policy to delete snapshots you no longer need.

A snapshot repository can safely scale to thousands of snapshots. However, to manage its metadata, a large repository requires more memory on the master node. Retention rules ensure a repository’s metadata doesn’t grow to a size that could destabilize the master node.

To take a snapshot without an SLM policy, use the create snapshot API. The snapshot name supports date math.

resp = client.snapshot.create(
    repository="my_repository",
    snapshot="<my_snapshot_{now/d}>",
)
print(resp)

response = client.snapshot.create(
  repository: 'my_repository',
  snapshot: '<my_snapshot_{now/d}>'
)
puts response

const response = await client.snapshot.create({
  repository: "my_repository",
  snapshot: "<my_snapshot_{now/d}>",
});
console.log(response);

# PUT _snapshot/my_repository/<my_snapshot_{now/d}>
PUT _snapshot/my_repository/%3Cmy_snapshot_%7Bnow%2Fd%7D%3E

Depending on its size, a snapshot can take a while to complete. By default, the create snapshot API only initiates the snapshot process, which runs in the background. To block the client until the snapshot finishes, set the wait_for_completion query parameter to true.

resp = client.snapshot.create(
    repository="my_repository",
    snapshot="my_snapshot",
    wait_for_completion=True,
)
print(resp)

response = client.snapshot.create(
  repository: 'my_repository',
  snapshot: 'my_snapshot',
  wait_for_completion: true
)
puts response

const response = await client.snapshot.create({
  repository: "my_repository",
  snapshot: "my_snapshot",
  wait_for_completion: "true",
});
console.log(response);

PUT _snapshot/my_repository/my_snapshot?wait_for_completion=true

You can also clone an existing snapshot using clone snapshot API.

To monitor any currently running snapshots, use the get snapshot API with the _current request path parameter.

resp = client.snapshot.get(
    repository="my_repository",
    snapshot="_current",
)
print(resp)

response = client.snapshot.get(
  repository: 'my_repository',
  snapshot: '_current'
)
puts response

const response = await client.snapshot.get({
  repository: "my_repository",
  snapshot: "_current",
});
console.log(response);

GET _snapshot/my_repository/_current

To get a complete breakdown of each shard participating in any currently running snapshots, use the get snapshot status API.

resp = client.snapshot.status()
print(resp)

response = client.snapshot.status
puts response

const response = await client.snapshot.status();
console.log(response);

GET _snapshot/_status

To get more information about a cluster’s SLM execution history, including stats for each SLM policy, use the get SLM stats API. The API also returns information about the cluster’s snapshot retention task history.

resp = client.slm.get_stats()
print(resp)

response = client.slm.get_stats
puts response

const response = await client.slm.getStats();
console.log(response);

GET _slm/stats

To get information about a specific SLM policy’s execution history, use the get SLM policy API. The response includes:

resp = client.slm.get_lifecycle(
    policy_id="nightly-snapshots",
)
print(resp)

response = client.slm.get_lifecycle(
  policy_id: 'nightly-snapshots'
)
puts response

const response = await client.slm.getLifecycle({
  policy_id: "nightly-snapshots",
});
console.log(response);

GET _slm/policy/nightly-snapshots

To delete a snapshot in Kibana, go to the Snapshots page and click the trash icon under the Actions column. You can also use the delete snapshot API.

resp = client.snapshot.delete(
    repository="my_repository",
    snapshot="my_snapshot_2099.05.06",
)
print(resp)

response = client.snapshot.delete(
  repository: 'my_repository',
  snapshot: 'my_snapshot_2099.05.06'
)
puts response

const response = await client.snapshot.delete({
  repository: "my_repository",
  snapshot: "my_snapshot_2099.05.06",
});
console.log(response);

DELETE _snapshot/my_repository/my_snapshot_2099.05.06

If you delete a snapshot that’s in progress, Elasticsearch cancels it. The snapshot process halts and deletes any files created for the snapshot. Deleting a snapshot doesn’t delete files used by other snapshots.

If you run Elasticsearch on your own hardware, we recommend that, in addition to backups, you take regular backups of the files in each node’s $ES_PATH_CONF directory using the file backup software of your choice. Snapshots don’t back up these files. Also note that these files will differ on each node, so each node’s files should be backed up individually.

By default, a snapshot that includes the cluster state also includes all feature states. Similarly, a snapshot that excludes the cluster state excludes all feature states by default.

You can also configure a snapshot to only include specific feature states, regardless of the cluster state.

To get a list of available features, use the get features API.

resp = client.features.get_features()
print(resp)

response = client.features.get_features
puts response

const response = await client.features.getFeatures();
console.log(response);

GET _features

The API returns:

{
  "features": [
    {
      "name": "tasks",
      "description": "Manages task results"
    },
    {
      "name": "kibana",
      "description": "Manages Kibana configuration and reports"
    },
    {
      "name": "security",
      "description": "Manages configuration for Security features, such as users and roles"
    },
    ...
  ]
}

To include a specific feature state in a snapshot, specify the feature name in the feature_states array.

For example, the following SLM policy only includes feature states for the Kibana and Elasticsearch security features in its snapshots.

resp = client.slm.put_lifecycle(
    policy_id="nightly-snapshots",
    schedule="0 30 2 * * ?",
    name="<nightly-snap-{now/d}>",
    repository="my_repository",
    config={
        "indices": "*",
        "include_global_state": True,
        "feature_states": [
            "kibana",
            "security"
        ]
    },
    retention={
        "expire_after": "30d",
        "min_count": 5,
        "max_count": 50
    },
)
print(resp)

const response = await client.slm.putLifecycle({
  policy_id: "nightly-snapshots",
  schedule: "0 30 2 * * ?",
  name: "<nightly-snap-{now/d}>",
  repository: "my_repository",
  config: {
    indices: "*",
    include_global_state: true,
    feature_states: ["kibana", "security"],
  },
  retention: {
    expire_after: "30d",
    min_count: 5,
    max_count: 50,
  },
});
console.log(response);

PUT _slm/policy/nightly-snapshots
{
  "schedule": "0 30 2 * * ?",
  "name": "<nightly-snap-{now/d}>",
  "repository": "my_repository",
  "config": {
    "indices": "*",
    "include_global_state": true,
    "feature_states": [
      "kibana",
      "security"
    ]
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 5,
    "max_count": 50
  }
}

Any index or data stream that’s part of the feature state will display in a snapshot’s contents. For example, if you back up the security feature state, the security-* system indices display in the get snapshot API's response under both indices and feature_states.

Some feature states contain sensitive data. For example, the security feature state includes system indices that may contain user names and encrypted password hashes. Because passwords are stored using cryptographic hashes, the disclosure of a snapshot would not automatically enable a third party to authenticate as one of your users or use API keys. However, it would disclose confidential information, and if a third party can modify snapshots, they could install a back door.

To better protect this data, consider creating a dedicated repository and SLM policy for snapshots of the cluster state. This lets you strictly limit and audit access to the repository.

For example, the following SLM policy only backs up the cluster state. The policy stores these snapshots in a dedicated repository.

resp = client.slm.put_lifecycle(
    policy_id="nightly-cluster-state-snapshots",
    schedule="0 30 2 * * ?",
    name="<nightly-cluster-state-snap-{now/d}>",
    repository="my_secure_repository",
    config={
        "include_global_state": True,
        "indices": "-*"
    },
    retention={
        "expire_after": "30d",
        "min_count": 5,
        "max_count": 50
    },
)
print(resp)

const response = await client.slm.putLifecycle({
  policy_id: "nightly-cluster-state-snapshots",
  schedule: "0 30 2 * * ?",
  name: "<nightly-cluster-state-snap-{now/d}>",
  repository: "my_secure_repository",
  config: {
    include_global_state: true,
    indices: "-*",
  },
  retention: {
    expire_after: "30d",
    min_count: 5,
    max_count: 50,
  },
});
console.log(response);

PUT _slm/policy/nightly-cluster-state-snapshots
{
  "schedule": "0 30 2 * * ?",
  "name": "<nightly-cluster-state-snap-{now/d}>",
  "repository": "my_secure_repository",
  "config": {
    "include_global_state": true,                 
    "indices": "-*"                               
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 5,
    "max_count": 50
  }
}

If you take dedicated snapshots of the cluster state, you’ll need to exclude the cluster state from your other snapshots. For example:

resp = client.slm.put_lifecycle(
    policy_id="nightly-snapshots",
    schedule="0 30 2 * * ?",
    name="<nightly-snap-{now/d}>",
    repository="my_repository",
    config={
        "include_global_state": False,
        "indices": "*"
    },
    retention={
        "expire_after": "30d",
        "min_count": 5,
        "max_count": 50
    },
)
print(resp)

const response = await client.slm.putLifecycle({
  policy_id: "nightly-snapshots",
  schedule: "0 30 2 * * ?",
  name: "<nightly-snap-{now/d}>",
  repository: "my_repository",
  config: {
    include_global_state: false,
    indices: "*",
  },
  retention: {
    expire_after: "30d",
    min_count: 5,
    max_count: 50,
  },
});
console.log(response);

PUT _slm/policy/nightly-snapshots
{
  "schedule": "0 30 2 * * ?",
  "name": "<nightly-snap-{now/d}>",
  "repository": "my_repository",
  "config": {
    "include_global_state": false,    
    "indices": "*"                    
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 5,
    "max_count": 50
  }
}

If you only use a single SLM policy, it can be difficult to take frequent snapshots and retain snapshots with longer time intervals.

For example, a policy that takes snapshots every 30 minutes with a maximum of 100 snapshots will only keep snapshots for approximately two days. While this setup is great for backing up recent changes, it doesn’t let you restore data from a previous week or month.

To fix this, you can create multiple SLM policies with the same snapshot repository that run on different schedules. Since a policy’s retention rules only apply to its snapshots, a policy won’t delete a snapshot created by another policy.

For example, the following SLM policy takes hourly snapshots with a maximum of 24 snapshots. The policy keeps its snapshots for one day.

resp = client.slm.put_lifecycle(
    policy_id="hourly-snapshots",
    name="<hourly-snapshot-{now/d}>",
    schedule="0 0 * * * ?",
    repository="my_repository",
    config={
        "indices": "*",
        "include_global_state": True
    },
    retention={
        "expire_after": "1d",
        "min_count": 1,
        "max_count": 24
    },
)
print(resp)

const response = await client.slm.putLifecycle({
  policy_id: "hourly-snapshots",
  name: "<hourly-snapshot-{now/d}>",
  schedule: "0 0 * * * ?",
  repository: "my_repository",
  config: {
    indices: "*",
    include_global_state: true,
  },
  retention: {
    expire_after: "1d",
    min_count: 1,
    max_count: 24,
  },
});
console.log(response);

PUT _slm/policy/hourly-snapshots
{
  "name": "<hourly-snapshot-{now/d}>",
  "schedule": "0 0 * * * ?",
  "repository": "my_repository",
  "config": {
    "indices": "*",
    "include_global_state": true
  },
  "retention": {
    "expire_after": "1d",
    "min_count": 1,
    "max_count": 24
  }
}

The following policy takes nightly snapshots in the same snapshot repository. The policy keeps its snapshots for one month.

resp = client.slm.put_lifecycle(
    policy_id="daily-snapshots",
    name="<daily-snapshot-{now/d}>",
    schedule="0 45 23 * * ?",
    repository="my_repository",
    config={
        "indices": "*",
        "include_global_state": True
    },
    retention={
        "expire_after": "30d",
        "min_count": 1,
        "max_count": 31
    },
)
print(resp)

const response = await client.slm.putLifecycle({
  policy_id: "daily-snapshots",
  name: "<daily-snapshot-{now/d}>",
  schedule: "0 45 23 * * ?",
  repository: "my_repository",
  config: {
    indices: "*",
    include_global_state: true,
  },
  retention: {
    expire_after: "30d",
    min_count: 1,
    max_count: 31,
  },
});
console.log(response);

PUT _slm/policy/daily-snapshots
{
  "name": "<daily-snapshot-{now/d}>",
  "schedule": "0 45 23 * * ?",          
  "repository": "my_repository",
  "config": {
    "indices": "*",
    "include_global_state": true
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 1,
    "max_count": 31
  }
}

The following policy creates monthly snapshots in the same repository. The policy keeps its snapshots for one year.

resp = client.slm.put_lifecycle(
    policy_id="monthly-snapshots",
    name="<monthly-snapshot-{now/d}>",
    schedule="0 56 23 1 * ?",
    repository="my_repository",
    config={
        "indices": "*",
        "include_global_state": True
    },
    retention={
        "expire_after": "366d",
        "min_count": 1,
        "max_count": 12
    },
)
print(resp)

const response = await client.slm.putLifecycle({
  policy_id: "monthly-snapshots",
  name: "<monthly-snapshot-{now/d}>",
  schedule: "0 56 23 1 * ?",
  repository: "my_repository",
  config: {
    indices: "*",
    include_global_state: true,
  },
  retention: {
    expire_after: "366d",
    min_count: 1,
    max_count: 12,
  },
});
console.log(response);

PUT _slm/policy/monthly-snapshots
{
  "name": "<monthly-snapshot-{now/d}>",
  "schedule": "0 56 23 1 * ?",            
  "repository": "my_repository",
  "config": {
    "indices": "*",
    "include_global_state": true
  },
  "retention": {
    "expire_after": "366d",
    "min_count": 1,
    "max_count": 12
  }
}