Elasticsearch Red Cluster Runbook: Lost Primaries and Recovery

A red cluster means at least one primary shard is not assigned, so some data cannot be searched and some writes are failing. The urgent instinct is to make it green. That instinct causes most of the permanent damage done during Elasticsearch incidents, because the fastest commands that turn a cluster green do it by discarding data. The right order is: confirm there is a master, find exactly which primaries are missing, find out why, and only then choose between waiting, fixing allocation, forcing a stale copy, restoring from snapshot or accepting an empty shard. This runbook gives that order, the commands at each step, what each recovery option costs in lost data, and how to tell people what is affected while you work.

What red actually means

Cluster health is the worst status of any index, and an index is red when any of its primaries is unassigned. That definition hides a wide range of severity:

• One primary of one old, rarely queried index, unassigned because a node is restarting. Minutes of partial results on a few queries.
• The write index of a data stream is red. Every write to that data stream fails until it is fixed.
• Primaries of many indices are gone because two nodes holding both copies of those shards were lost together. That is real data loss unless the nodes come back or a snapshot exists.

What clients see while a primary is unassigned:

• Searches that touch the index still run against the shards that are available. By default allow_partial_search_results is true, so the response succeeds with _shards.failed greater than 0. Applications and dashboards that do not check _shards.failed show wrong numbers without any error. This is the most underestimated impact of a red cluster.
• Index, update and delete requests routed to the missing shard wait for the primary (the default request timeout is one minute) and then fail with an unavailable_shards_exception. In a bulk request, only the items routed to that shard fail; the rest succeed. Clients that retry whole bulk requests will create duplicates for the items that succeeded unless they use their own document IDs.

A different kind of red: no elected master. Then GET _cluster/health does not return red, it returns an error, typically master_not_discovered_exception. Nothing can be allocated without a master, so this comes first.

Triage order

Work in this order. Each step either resolves the incident or narrows it.

1. Is there an elected master?
2. Which indices and which primaries are red?
3. Is a node missing, and is it coming back?
4. What does allocation explain say about each missing primary?
5. Is any copy of the data still on disk somewhere?
6. Is there a recent snapshot?
7. Decide, write the decision down, then act.

Step 1: master

GET _cat/master?v

GET _cat/nodes?v&h=name,node.role,master,heap.percent,disk.used_percent,uptime

If these fail with a master-not-discovered error, the incident is about quorum, not shards. Check how many master-eligible nodes are running, check their logs for discovery and election messages, and restore enough of them to form a majority of the voting configuration. Do not run any shard-level recovery commands until a master is elected. See /guides/elasticsearch-cluster-topology/ for how the voting configuration works and why quorum was lost.

Step 2: scope

GET _cluster/health?filter_path=status,number_of_nodes,number_of_data_nodes,active_primary_shards,unassigned_shards,delayed_unassigned_shards,initializing_shards

GET _cat/indices?v&health=red&h=index,health,status,pri,rep,docs.count,store.size

GET _cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason,unassigned.at,unassigned.details&s=state

From _cat/shards, list every row where prirep is p and state is UNASSIGNED. That is the exact set of lost primaries. Write it down: index, shard number, unassigned.reason, unassigned.at.

In recent 8.x releases the health report adds a diagnosis per indicator, including shard availability, and is worth reading alongside:

GET _health_report

Its shards_availability indicator lists affected indices and suggests a cause.

For each red index, decide its weight: is it a data stream's current write index? (GET _data_stream/<name> shows the backing indices, the last one is the write index.) Is it business-critical or a 400-day-old log index?

Step 3: missing nodes

Compare number_of_data_nodes with what you expect. If unassigned.reason is NODE_LEFT for the red primaries, the copies were on nodes that left. Then the most important question of the incident is: will those nodes come back with their disks intact?

• If yes (a restart, a host reboot, a network partition that heals): bring them back. The primaries will be allocated to the returning nodes from their on-disk copies, with no data loss. This is almost always the best outcome and is worth waiting for.
• If the disks are gone (instance terminated, volume destroyed): the data on those nodes is gone, and the remaining options are below.

Note that a primary only goes red when no in-sync copy is available. With one replica, that means both the primary's node and the replica's node are gone, or the replica was not in sync. Two simultaneous node losses in a cluster where both copies of some shards lived on those two nodes is a common cause; zone awareness exists to prevent exactly that.

Step 4: allocation explain

GET _cluster/allocation/explain
{
  "index": "orders-2026.10",
  "shard": 3,
  "primary": true
}

The two answers that matter for a red primary:

can_allocate: no with deciders that say NO. There is a valid copy on disk, but the allocator is not allowed to put the primary there: a filter, a disk watermark, a tier preference, an allocation enable setting. This is an allocation problem, not data loss. Fix the decider, and the primary is allocated. Every common decider and its fix is in /guides/elasticsearch-unassigned-shards/. New indices that are red from creation (INDEX_CREATED) are always this case.

can_allocate: no_valid_shard_copy. The allocator cannot find a valid, in-sync copy of the primary on any node in the cluster. The explanation says, in some form, that a previous copy existed but can no longer be found, or that the copies found are stale or corrupt. Every option from here on loses data, except bringing back the node that held the in-sync copy.

can_allocate: awaiting_info means the master is still asking nodes what they hold, usually right after nodes join. Wait a few seconds and run it again.

Step 5: what is on disk

The index shard stores API lists every copy of each shard that nodes report, whether in sync or not:

GET orders-2026.10/_shard_stores?status=red

{
  "indices": {
    "orders-2026.10": {
      "shards": {
        "3": {
          "stores": [
            {
              "Ab12...": { "name": "data-7", "transport_address": "10.0.4.17:9300" },
              "allocation_id": "kQ9...",
              "allocation": "unused"
            }
          ]
        }
      }
    }
  }
}

For each store entry:

• allocation is primary, replica or unused. unused means the node holds a copy that is not one of the in-sync allocation IDs the cluster expects: a stale copy.
• store_exception, if present, means the copy could not be opened, often because of corruption.

If you see a store with no exception, you have a stale copy that could be forced into service. If you see no stores at all, or only stores with exceptions, there is nothing on disk to use.

Step 6: snapshots

GET _cat/snapshots/nightly-s3?v&s=end_epoch:desc

GET _snapshot/nightly-s3/*?sort=start_time&order=desc&size=5

Find the most recent successful snapshot that contains the red index. A snapshot with state PARTIAL may be missing some shards; check that the shard you need was included successfully:

GET _snapshot/nightly-s3/nightly-2026.10.21/_status

Snapshot management and retention are covered in /guides/elasticsearch-snapshot-restore-dr/.

The recovery options, and what each one costs

For a primary with no_valid_shard_copy, there are five possible outcomes. Rank them by data preserved.

allocate_stale_primary

This tells the master to use a stale copy as the primary. You must name the node holding it and explicitly accept data loss:

POST _cluster/reroute
{
  "commands": [
    {
      "allocate_stale_primary": {
        "index": "orders-2026.10",
        "shard": 3,
        "node": "data-7",
        "accept_data_loss": true
      }
    }
  ]
}

What you lose: every operation acknowledged to clients after that copy stopped being in sync. The cluster does not record how many that is. You can bound it: the copy fell out of sync around when its node left or failed (look for that event in the master logs, or the unassigned.at time from an earlier failure), and everything indexed into that shard since then is gone.

The other consequence: if a node holding a newer, in-sync copy rejoins later, that newer data is discarded in favor of the forced stale primary. So before running this command, be certain the in-sync copy is not coming back. That is why Step 3 comes first.

After a stale primary is allocated, replicas are rebuilt from it as normal.

allocate_empty_primary

This creates a brand-new, empty shard in place of the lost one:

POST _cluster/reroute
{
  "commands": [
    {
      "allocate_empty_primary": {
        "index": "orders-2026.10",
        "shard": 3,
        "node": "data-2",
        "accept_data_loss": true
      }
    }
  ]
}

What you lose: every document that lived in that shard. With 12 primaries, that is roughly one twelfth of the index, scattered across every time range and every entity, because documents are routed to shards by a hash of their ID (or routing value). The index then looks complete and is silently missing documents. As with a stale primary, if any copy of the old data later appears on a rejoining node, it is deleted.

Use it when the index is expendable (for example, a log index you will reingest, or one you are about to delete anyway but need writable today), and record which shard was emptied, when, and by whom.

Restoring from snapshot

A restore brings back the index exactly as it was at snapshot time. You cannot restore over an open index, so there are two approaches.

Restore under a new name, then switch. This is the safe default, because the red index stays in place until you have verified the restore:

POST _snapshot/nightly-s3/nightly-2026.10.21/_restore
{
  "indices": "orders-2026.10",
  "include_global_state": false,
  "include_aliases": false,
  "rename_pattern": "(.+)",
  "rename_replacement": "restored-$1",
  "index_settings": { "index.number_of_replicas": 0 }
}

Restoring with zero replicas finishes faster; add replicas after it is green. include_aliases: false matters here: by default a restore also restores the index's aliases, and the restored copy would join the orders alias next to the red index before you have verified it. Watch progress:

GET _cat/recovery/restored-orders-2026.10?v&active_only=true
GET _cluster/health/restored-orders-2026.10?wait_for_status=yellow&timeout=30m

Then verify, move the alias, and set replicas:

GET restored-orders-2026.10/_count

POST _aliases
{
  "actions": [
    { "remove": { "index": "orders-2026.10", "alias": "orders" } },
    { "add": { "index": "restored-orders-2026.10", "alias": "orders" } }
  ]
}

PUT restored-orders-2026.10/_settings
{ "index.number_of_replicas": 1 }

Delete the red index only once the restored one is verified and serving.

Delete, then restore under the original name. Faster and simpler when nothing points at the index by alias, but there is a window with no index at all, and if the restore fails you have nothing. Use it only when the red index is beyond saving anyway.

Restore time arithmetic, with stated assumptions: a 600 GB index, restored from object storage at an effective 400 MB/s across the receiving nodes, takes 600,000 / 400 = 1,500 seconds, about 25 minutes, before replicas. Measure your own restore throughput in a rehearsal, because repository throughput, indices.recovery.max_bytes_per_sec and the number of receiving nodes all limit it.

Data streams

The write index of a data stream cannot be deleted while it is the write index, and you usually cannot wait to restore it because every write is failing. Restore ingest first:

POST logs-payments-default/_rollover

Rollover creates a new, empty backing index and makes it the write index. As long as the new index can be allocated, writes resume immediately. The red index is now an ordinary backing index, and you can deal with it without pressure: restore it under a new name, then, once verified, remove the red backing index from the data stream and add the restored one with the modify data stream API:

POST _data_stream/_modify
{
  "actions": [
    { "remove_backing_index": { "data_stream": "logs-payments-default", "index": ".ds-logs-payments-default-2026.10.21-000118" } },
    { "add_backing_index": { "data_stream": "logs-payments-default", "index": "restored-.ds-logs-payments-default-2026.10.21-000118" } }
  ]
}

Elastic documents add_backing_index as an expert-level action that can cause improper data stream behavior if misused, so confirm the restored index carries the data stream's mappings (including @timestamp) before you add it. The write index can never be removed, which is one more reason the rollover comes first.

Rollover first, restore second, is the most useful single habit in data stream incidents.

Corruption

If allocation explain or _shard_stores shows a corruption exception for the only copy, the elasticsearch-shard command-line tool can remove the corrupted parts of a shard (the remove-corrupted-data subcommand) on a stopped node, after which the shard can be allocated as a stale primary. It deletes whatever data was in the corrupted segments. Prefer restoring from snapshot; use the tool only when there is no snapshot and partial data beats none. Also investigate the cause: corruption on a healthy filesystem is rare, and the disk or controller that caused it may still be in service.

Deciding: a worked example

Situation: orders-2026.10 has 12 primaries and 1 replica. Two nodes in the same rack were lost at 09:40 when their host failed; their disks are on local NVMe and the host will not be repaired today. Shards 3 and 9 had both copies on those two nodes and are red. data-7 dropped out of the cluster at 08:00 after a failed disk check, and the replica of shard 3 it held was rebuilt on one of the two nodes that later failed. data-7 rejoined at 09:55, and _shard_stores now shows its old copy of shard 3 as unused. There is no copy of shard 9 anywhere. The last nightly snapshot completed at 01:00.

Options for shard 3:

• Restore from snapshot: lose writes from 01:00 to 09:40, 8 h 40 min.
• Stale primary from data-7: that copy stopped receiving writes when data-7 left at 08:00, so it is missing everything written to shard 3 from 08:00 to 09:40, 1 h 40 min.
• Stale primary loses less, but the loss is unknowable in detail. Restore loses more, but you know exactly what is missing: everything after 01:00, which the application team may be able to replay from their own system.

Options for shard 9: restore from snapshot (lose 8 h 40 min) or empty primary (lose everything in that shard). Restore.

Mixed answers in one index are awkward because a restore works per index, not per shard. A defensible plan: restore the whole index under a new name from the 01:00 snapshot; ask the application team to replay orders from 01:00 into the restored index; switch the alias; delete the red index. That loses nothing in the end if the replay is complete. If no replay source exists, the stale primary for shard 3 plus a snapshot-based backfill of shard 9's documents is more work and is rarely worth it.

The point of the example: the decision depends on facts outside Elasticsearch (is there a source of truth? how valuable is each hour?). That is why you write the decision down and get it agreed, instead of typing allocate_empty_primary at 3 a.m.

Communicating impact

Send the first message within minutes, even with partial knowledge, and update it at every decision. A template:

Status: Elasticsearch cluster prod-search is RED since 09:40 UTC.
Affected: index orders-2026.10, 2 of 12 shards unavailable.
  - Searches over orders return partial results (about 1/6 of documents missing, all dates).
  - Writes for about 1/6 of new orders are failing; clients see errors.
Not affected: all other indices; logging; customer search.
Cause: two data nodes lost with their disks in a host failure at 09:40.
Plan: restore orders-2026.10 from the 01:00 snapshot (ETA 30 min), then replay
  orders from 01:00 from the order service. Expected final data loss: none,
  if replay completes.
Next update: 10:30 UTC or at next decision.

Three things to state explicitly every time: what is wrong for users right now (partial results are the one people miss), whether data is lost or only unavailable, and what decision you are making and its data cost.

Anti-patterns

Full cluster restart to "reset" things. It turns one red index into a cluster recovering every shard, and if the cause was quorum or disk, it is still there afterwards.

allocate_empty_primary before checking whether the node is coming back. If the node returns, its good data is deleted in favor of your empty shard.

Setting number_of_replicas: 0 on a red index. It removes the redundancy of the shards that are still healthy and does nothing for the missing primary.

Deleting data directories on a node to "clean" it. That destroys the stale or in-sync copies that might be the only remaining data.

Ignoring _shards.failed in clients. Partial results without warning is the user-visible failure of a red cluster. Make critical applications check it, or set allow_partial_search_results to false on requests where a partial answer is worse than an error.

Running retry_failed and walking away. It helps when allocation failed for a transient reason. It cannot create a copy that does not exist.

Runbook checklist

1. GET _cat/master?v. No master: restore quorum first. Stop here until a master is elected.
2. GET _cluster/health, GET _cat/indices?v&health=red, and _cat/shards filtered to unassigned primaries. Write down every red primary.
3. If a data stream write index is red: POST <data-stream>/_rollover to restore ingest.
4. Send the first impact message: affected indices, partial results, failing writes.
5. Check for missing nodes. If they will return with disks intact, bring them back and wait.
6. GET _cluster/allocation/explain per red primary. If deciders say NO, fix the decider (see /guides/elasticsearch-unassigned-shards/).
7. If no_valid_shard_copy: GET <index>/_shard_stores?status=red for stale copies, and list snapshots containing the index.
8. Compare data loss per option: snapshot age versus how long the stale copy has been out of sync, and whether a source of truth can replay the gap.
9. Get the decision agreed and recorded with its data cost.
10. Act: restore under a new name (preferred), or allocate_stale_primary, or, for expendable data only, allocate_empty_primary, each with accept_data_loss: true stated knowingly.
11. Verify counts, switch aliases or modify the data stream, add replicas, confirm green with GET _cluster/health?wait_for_status=green&timeout=10m.
12. Afterwards: fix why both copies could be lost together (zone or rack awareness), and check snapshot frequency against how much loss the business accepted today.

Key takeaways

• Red means an unassigned primary: partial search results that clients often do not notice, and failed writes for the affected shard.
• Triage in order: master, scope, missing nodes, allocation explain, shard stores, snapshots, then decide.
• can_allocate: no with deciders is an allocation problem with no data loss; no_valid_shard_copy is the data-loss branch.
• allocate_stale_primary loses every write since the copy went stale and discards a newer copy if it returns; allocate_empty_primary loses the whole shard.
• Restore under a new name and switch the alias; for data streams, roll over first to restore ingest.
• Write down the decision and its data cost before acting, and tell users about partial results explicitly.

How Sparced helps

When the cluster is red, Sparced makes its follow-up calls, including _cluster/allocation/explain for each stuck shard, so every unassigned primary is listed with the cluster's own reason, distinguishing a decider refusal from a primary with no valid copy. It reads nodes, shards, allocation and snapshots in the same pass, so the missing nodes and the snapshot repositories and their recent snapshots appear alongside the red indices. Each finding gives a fix command with your index, shard and node names filled in, and a check to confirm the result. It writes nothing to the cluster and never runs a reroute for you; choosing between a stale primary, an empty primary and a restore stays with you.