Elasticsearch Snapshots, SLM and Disaster Recovery Planning

Replicas protect you from a node failing. They do not protect you from a deleted index, a bad mapping change pushed to every index, a ransomware wipe, a region outage or a cluster whose metadata is corrupt. Snapshots do, and in Elasticsearch they are the only supported way to back up a cluster: copying a data directory is not a backup. The answer this article builds is a recovery plan with numbers in it: a repository you have verified, an SLM policy whose schedule sets your recovery point objective, a retention rule that keeps enough history to recover from a mistake noticed late, a restore time you have measured rather than guessed, and a monthly restore test that proves the whole chain works.

The mental model: snapshots are segment-level and incremental

A snapshot is not a dump of documents. Elasticsearch shards are Lucene indices made of immutable segment files. When a snapshot runs, each primary shard copies the segment files that make up its current state into the repository, but only the files the repository does not already hold. Files that earlier snapshots already uploaded are referenced, not copied again.

Three properties follow:

• Every snapshot is logically full. Each snapshot can be restored on its own; there is no "base plus increments" chain to replay. Deleting an older snapshot only removes files no remaining snapshot references.
• Cost depends on segment churn, not on index size. An index that is not being written to costs almost nothing in a second snapshot. A heavily written index whose segments are constantly merged costs close to its full size, because merges produce new files.
• Force merge resets the increment. After a force merge to one segment, the next snapshot uploads the whole shard again, since every file is new. For indices on ILM that force merge in warm, the warm transition is followed by a full-size upload of that index. Plan repository bandwidth for it.

A snapshot also captures metadata. With include_global_state: true it includes the cluster state: persistent cluster settings, index templates, component templates, ingest pipelines, ILM policies and stored scripts. Separately, feature states capture the system indices owned by a feature, such as security (users, roles, API keys), Kibana (saved objects), and Watcher. You can list them:

GET _features

Snapshots are taken from primaries and are consistent per shard, not across the whole cluster at one instant. Two indices in the same snapshot may be captured seconds apart. For logs and metrics that is irrelevant; for an application that needs two indices consistent with each other, pause writes during the snapshot or design the application to tolerate it.

Repositories: types and how to register one

A typical S3 registration, with credentials stored in the keystore on every node rather than in the request:

bin/elasticsearch-keystore add s3.client.default.access_key
bin/elasticsearch-keystore add s3.client.default.secret_key

Then reload secure settings with POST _nodes/reload_secure_settings, and register the repository:

PUT _snapshot/prod-s3
{
  "type": "s3",
  "settings": {
    "bucket": "acme-es-snapshots",
    "base_path": "prod-eu-1",
    "client": "default"
  }
}

If the nodes run with an IAM role or workload identity, you can omit the keys and the client uses the environment's credentials.

For fs, the mount must be the same shared filesystem on every master-eligible and data node, at the same path, and the path must appear in path.repo in elasticsearch.yml. A local disk on each node that happens to have the same path is not a shared filesystem, and snapshots taken that way are not restorable.

Verify, then analyze

Registration verifies that nodes can write to the repository by default. You can re-verify any time:

POST _snapshot/prod-s3/_verify

The response lists the nodes that succeeded. A node missing from the list cannot reach the repository, and shards whose primaries live on it will fail to snapshot.

Verification proves connectivity, not correctness. The repository analysis API exercises the storage the way snapshots do, including concurrent writes, reads of the same blobs from many nodes, and large blobs, and reports failures and timings. It is the right tool for an S3-compatible store from a third party or for a NAS, where subtle consistency bugs exist:

POST _snapshot/prod-s3/_analyze?timeout=1h

Elastic's guidance is to run a first analysis with the default parameters to catch simple problems, then run increasingly large ones up to at least blob_count=2000, max_blob_size=2gb, max_total_data_size=1tb and register_operation_count=100, always with a generous timeout (an hour or more), on a multi-node cluster of similar size to production:

POST _snapshot/prod-s3/_analyze?blob_count=2000&max_blob_size=2gb&max_total_data_size=1tb&register_operation_count=100&timeout=2h

It consumes real network bandwidth, storage and I/O on the repository, so run it in a quiet period. A common precaution is to point it at a repository registered on a separate base path rather than the one holding your production snapshots. A storage system that fails analysis is not safe to use as a repository even if snapshots appear to succeed.

One writer per repository

Only one cluster should ever write to a given repository path. If a second cluster, such as a DR cluster or a staging cluster, needs to read the same snapshots, register it there with "readonly": true. Two clusters writing to the same repository can corrupt its metadata, which would make every snapshot in it unreadable. This is the single most damaging snapshot mistake and it is easy to make during a migration.

Taking a snapshot manually

PUT _snapshot/prod-s3/pre-upgrade-2026.10.21?wait_for_completion=false
{
  "indices": "*,-.*",
  "ignore_unavailable": true,
  "include_global_state": true,
  "feature_states": ["security", "kibana"],
  "metadata": { "reason": "before 9.x upgrade", "taken_by": "jsmith" }
}

Check progress and the result:

GET _snapshot/prod-s3/pre-upgrade-2026.10.21/_status

GET _snapshot/prod-s3/pre-upgrade-2026.10.21

The snapshot state is one of IN_PROGRESS, SUCCESS, PARTIAL, FAILED or INCOMPATIBLE. PARTIAL means some shards were not captured; the failures array says which and why. By default a snapshot fails if any included index has an unavailable primary. Setting "partial": true allows a snapshot of a cluster with red indices, capturing what can be captured. That is useful during an incident, but a PARTIAL snapshot from SLM every night is a silent data protection gap.

SLM: scheduled snapshots with retention

Snapshot lifecycle management runs snapshots on a cron schedule and deletes old ones according to a retention rule.

PUT _slm/policy/nightly
{
  "schedule": "0 30 1 * * ?",
  "name": "<nightly-{now/d}>",
  "repository": "prod-s3",
  "config": {
    "indices": ["*"],
    "include_global_state": true,
    "feature_states": ["security", "kibana"]
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 7,
    "max_count": 60
  }
}

The schedule uses Elasticsearch's cron syntax with a seconds field first, evaluated in UTC. The name supports date math, so snapshots are named per day; SLM appends a unique suffix so that two runs on the same day do not collide.

Retention semantics:

• expire_after: snapshots older than this become eligible for deletion.
• min_count: always keep at least this many, even if they are older than expire_after. This protects you when snapshots have been failing: without it, retention would happily delete your last good snapshots while new ones fail.
• max_count: never keep more than this many, even if they are younger than expire_after.

Retention does not run at snapshot time. It runs on its own cluster-wide schedule, slm.retention_schedule, which defaults to daily at 01:30 UTC. Retention only considers snapshots taken by that policy; snapshots you take by hand are never touched by it.

Run a policy now and check its health:

POST _slm/policy/nightly/_execute

GET _slm/policy/nightly

The response carries last_success, last_failure (with the error), next_execution, and statistics. A last_failure newer than last_success is the alert condition. GET _slm/stats gives cluster-wide counts of snapshots taken, failed and deleted.

Layering policies

A single policy rarely fits every need. A common layout:

Because snapshots are incremental, the hourly policy costs mostly the churn since the last hour, not the cluster size. The hourly policy can exclude large, easily rebuilt indices to keep it cheap.

Pair the delete phase of ILM with SLM using the wait_for_snapshot action in the delete phase. ILM then waits until the named SLM policy has taken a snapshot that includes the index before deleting it, which guarantees the index exists somewhere before it goes.

RPO and RTO arithmetic

Recovery point objective is how much data you can lose. Recovery time objective is how long you can be down. Both have to be computed, not stated.

RPO

Worst-case data loss from snapshots alone is the snapshot interval plus the duration of the snapshot, because data written during an in-progress snapshot may not be in it.

Example: hourly snapshots that take 8 minutes. Worst case RPO is about 68 minutes. If the requirement is 15 minutes, hourly snapshots cannot meet it; you need either a shorter interval (which works only if snapshots finish in well under the interval) or a different mechanism. Cross-cluster replication typically lags by seconds and is the standard answer for tight RPO; see cross-cluster replication and search. For logging, the data source often provides its own RPO: if Kafka retains 72 hours of events, you can replay from it after a restore, and as long as Kafka still holds everything written since the snapshot, your effective RPO for that data is close to zero. See logging with a Kafka buffer.

RTO

Restore time is dominated by moving bytes. A restore copies primary shard data from the repository to the cluster, then builds replicas by peer recovery from those primaries. Both are throttled per node by indices.recovery.max_bytes_per_sec, which defaults to 40mb per node in most configurations (dedicated cold and frozen nodes get a higher default in recent versions). Concurrency is also limited by how many recoveries a node runs at once.

Worked example with stated assumptions:

• Data to restore: 6 TB of primaries, one replica wanted.
• 12 data nodes, recovery throttle at the default 40 MB/s per node.
• Repository and network can sustain more than the aggregate throttle.

1. Aggregate restore rate: 12 x 40 MB/s = 480 MB/s.
2. Primary restore: 6,000,000 MB / 480 MB/s = 12,500 s, about 3.5 hours.
3. Replica recovery: another 6 TB across the cluster at the same throttle, about 3.5 hours more, though you can serve traffic from primaries alone during this period if you accept no redundancy.
4. Add cluster provisioning (if the cluster itself was lost), client cutover and validation. Assume 1 hour.
5. RTO: about 4.5 hours to serve, about 8 hours to full redundancy.

Raising the throttle is the obvious lever, and during a DR restore it is appropriate if the disks and network can take it:

PUT _cluster/settings
{
  "persistent": { "indices.recovery.max_bytes_per_sec": "200mb" }
}

At 200 MB/s per node, step 2 becomes 6,000,000 / 2,400 = 2,500 s, about 42 minutes, provided the repository actually delivers 2.4 GB/s. Object stores usually scale with parallelism, but a NAS often does not; measure with the analysis API or a test restore rather than assuming. Remember to set the throttle back afterwards.

The second lever is restoring less. Restore the last 7 days of a 90-day logging cluster first, bring clients back, then restore the rest in the background. A third lever is to set index.number_of_replicas: 0 in the restore request and add replicas later, which delivers searchable data in half the transfer.

The last row matters. Replication of any kind copies mistakes. Only point-in-time snapshots let you go back to before the mistake.

Restore strategies

The restore API restores some or all indices from a snapshot into a running cluster:

POST _snapshot/prod-s3/nightly-2026.10.20-abc123/_restore
{
  "indices": "orders-*",
  "ignore_unavailable": true,
  "include_global_state": false,
  "include_aliases": false,
  "rename_pattern": "(.+)",
  "rename_replacement": "restored-$1",
  "index_settings": { "index.number_of_replicas": 0 },
  "ignore_index_settings": ["index.refresh_interval"]
}

Rules to remember:

• You cannot restore over an open index of the same name. Close or delete it first, or rename on restore. Renaming is safer: it lets you compare old and restored side by side.
• include_global_state: false is the default, and it should stay that way unless you are rebuilding a cluster from scratch. Restoring global state overwrites persistent settings, templates, pipelines and ILM policies on the target cluster.
• Feature states restore the system indices of a feature in full, replacing what is there. Restoring the security feature state replaces every user, role and API key in the cluster. On a running cluster, that can lock out the people doing the restore.
• Data streams are restored with their backing indices. The data stream needs a matching template with data_stream enabled on the target cluster for writes to work after restore, so restore or create templates first.
• index_settings overrides settings on the restored indices; ignore_index_settings drops settings from the snapshot. Use these to drop allocation filters that name nodes that do not exist in the target cluster, which would otherwise leave restored shards unassigned.

Three scenarios

Recover a deleted index. Restore it alone from the latest snapshot that contains it, with its original name since it no longer exists. Then reconcile any writes between the snapshot and the deletion from the source.

Undo a bad bulk update. Restore the index renamed (restored-orders-2026), validate it, then switch the application's alias from the damaged index to the restored one. Delete the damaged index only after the switch is confirmed. Aliases make this a one-call cutover; see zero-downtime reindexing.

Rebuild a lost cluster. Build a new cluster on a compatible version, register the repository (read-only first, until you are sure the old cluster is gone, then writable), restore global state and feature states first, then indices, newest first. Raise the recovery throttle during the restore.

Version compatibility

You can restore a snapshot into a cluster of the same version or a newer one. As a regular index, an index restores into a cluster whose major version is at most one higher than the version that created it: an index created in 8.x restores into 9.x, but an index created in 7.x does not restore into 9.x as a normal index. Elastic's compatibility table allows a 7.x index into 9.x only in read-only form, as an archive index or mounted as a searchable snapshot; indices from 5.x and 6.x reach 8.3 and later only as archive indices. To keep full functionality, reindex older indices on an intermediate version before the upgrade. Restoring into an older version is not supported. Check the creation version of your oldest indices before any major upgrade; see rolling upgrades.

Testing restores

A snapshot that has never been restored is a hypothesis. A monthly test, against a separate cluster that registers the repository as read-only, turns it into evidence:

1. Pick the latest SLM snapshot and a random older one within retention.
2. Restore a representative set: one large index, one data stream, and the security and Kibana feature states (on the test cluster only).
3. Time each restore and compute the observed MB/s per node. Compare it with the figure in your RTO calculation, and update the calculation.
4. Validate: GET <index>/_count matches the count recorded at snapshot time (record counts in snapshot metadata or from monitoring), a few known documents are retrievable by id, a saved Kibana dashboard opens, a test user can log in.
5. Delete the test cluster's restored data.
6. Record the date, snapshot name, durations and any failures.

The check that most often fails is not the data, it is everything around it: the repository credentials exist only on the old cluster, the templates for a data stream were never in a snapshot, or the restore takes three times longer than anyone assumed.

Failure modes and anti-patterns

Snapshots failing silently. Symptom: no one notices until a restore is needed. Confirm with GET _slm/policy and last_failure. Fix: alert when last_success is older than twice the schedule interval.

Repository unreachable from one node. Symptom: PARTIAL snapshots, with failures naming shards on the same node. Confirm with POST _snapshot/<repo>/_verify. Fix: that node's network path, credentials or mount.

Retention deleting the last good snapshots. Symptom: after a week of failures, only recent failed attempts remain. Fix: always set min_count.

Snapshotting a single huge index constantly. Symptom: every snapshot uploads hundreds of gigabytes. The index is force merged regularly, or receives heavy updates causing merges. Confirm with snapshot _status stats (incremental file counts and sizes). Fix: snapshot that index less often, or stop force merging it.

Using the same repository from two clusters. Symptom: repository is corrupted or concurrent modification errors. Fix: one writer only; others readonly: true. If corruption has already happened, register a fresh base path and take new snapshots immediately, then investigate what remains readable.

Too many snapshots. A repository with thousands of snapshots makes listing and deletion slow and increases master work. Fix: retention with max_count, and layered policies rather than one very frequent policy with long retention.

Restoring global state into a live cluster. Symptom: templates, ILM policies or settings revert after a restore. Fix: restore with include_global_state: false unless rebuilding.

Runbook: restore under pressure

1. Stop the bleeding. If writes are causing the damage, stop the producer or block the index (PUT <index>/_block/write).
2. Pick the snapshot. GET _snapshot/<repo>/*?verbose=false lists names and times; choose the newest one taken before the damage. Confirm its state is SUCCESS and that it contains the indices: GET _snapshot/<repo>/<snapshot>.
3. Check the target. Disk headroom for the restored data on the target nodes, below the high watermark (90% by default); versions compatible; templates present for any data stream.
4. Raise the throttle if speed matters and hardware allows: indices.recovery.max_bytes_per_sec.
5. Restore renamed, with zero replicas, unless the original is gone and the name must be the same.
6. Watch progress. GET _cat/recovery?active_only=true&v shows bytes and percentage per shard; GET _cluster/health/<index> shows when it is green or yellow.
7. Validate counts and sample documents.
8. Cut over with an alias swap, then add replicas.
9. Reset the throttle, and write down the observed restore rate for the next RTO estimate.

Key takeaways

• Snapshots are incremental at the segment-file level and each one is logically full; cost tracks segment churn, and force merge resets it.
• Verify every repository, run repository analysis on anything that is not a major cloud object store, and never let two clusters write to the same repository.
• SLM sets your RPO through its schedule and protects history through min_count; alert on last_failure.
• RTO is bytes divided by aggregate recovery throughput, twice if you rebuild replicas, plus provisioning. Measure it with real restores.
• Replication copies mistakes. Only snapshots let you go back in time.
• Restore renamed, without global state, then cut over with an alias.

How Sparced helps

Sparced reads the cluster's snapshot state as part of its single pass, so failed or partial snapshots and repository problems show up as findings with the snapshot and repository names and the cluster's own error text. It reads shards, allocation and disk in the same pass, which tells you whether a restore target has the headroom for the data before you start. When restored shards are stuck unassigned, for example because of an allocation filter carried over from the snapshot, its allocation explain calls name the decider and the shard, and each finding carries a fix command with the names filled in and a check to confirm it. It does not take, delete or restore snapshots, and writes nothing to the cluster.