Elasticsearch Disk Watermarks, Flood Stage and Headroom Planning

When an Elasticsearch data node runs low on disk, the cluster does three things in sequence: it stops putting new shards on that node, then it tries to move shards off it, and finally it blocks writes to every index with a shard on it. Those three steps are the low, high and flood stage disk watermarks, at 85%, 90% and 95% by default, with absolute "max headroom" caps that matter on large disks. The incident usually looks like "indexing suddenly failed with index read-only / allow delete". The fix is almost never to raise the watermarks. It is to free or add disk, let the block release itself, and then plan headroom so the cluster can lose a node and still have room to move its shards. This article explains how each watermark is evaluated, what the block does, the arithmetic for headroom, and a runbook for a disk-full incident.

The mental model: the disk threshold decider

Shard placement in Elasticsearch is decided by a set of allocation deciders. Each one answers "may this shard go on (or stay on) this node?" with YES, NO, or THROTTLE. The disk_threshold decider is the one that reads disk usage.

The master learns disk usage from a periodic poll of every node's filesystem statistics, controlled by cluster.info.update.interval (default 30s). This has two practical consequences:

• Disk decisions lag reality by up to the polling interval. A bulk load that writes fast can push a node well past a watermark between polls.
• The decider sees the filesystem, not just Elasticsearch's data. Logs, heap dumps, other processes and the filesystem's own reserved blocks all count, because the decider looks at total and available bytes on the data path.

The decider also accounts for shards that are being relocated onto a node, by subtracting their expected size from available space, so it does not overcommit a node while recoveries are in flight.

The whole mechanism is controlled by cluster.routing.allocation.disk.threshold_enabled, which defaults to true. Turning it off removes the only thing standing between a filling disk and a node that cannot write at all. Do not.

The three watermarks and what each one does

All three are dynamic cluster settings.

Low: stop adding

The low watermark is a soft fence. Existing shards stay where they are and keep accepting writes. But the decider says NO to allocating any new replica or relocated shard to the node. On a cluster where every node is past 85%, new replicas go unassigned and the cluster turns yellow. The allocation explain API will say so plainly, naming the node's disk usage (see unassigned shards).

The exemption for primaries of new indices exists so that a rollover or a new daily index can still be created. That exemption ends at the high watermark.

High: move away

At the high watermark the decider starts saying NO to remaining on the node, and the allocator plans relocations to nodes that have room. Relocation copies the shard over the network and only removes the source once the copy is complete, so disk on the source node does not fall until each relocation finishes. On a node that is filling quickly, relocation may not keep up.

If no other node has room below its own low watermark, nothing moves. The node stays above 90% and keeps filling.

Flood stage: stop writing

At flood stage, the cluster applies index.blocks.read_only_allow_delete to every index that has a shard on that node. Note the scope: it is every index with a shard on the node, not just the shards on it. One full node can block writes to dozens of indices whose other shards are on healthy nodes.

The block does what its name says: the index becomes read-only, except that deleting the whole index is still allowed, so you can free space. Index writes fail with an error like:

{
  "error": {
    "type": "cluster_block_exception",
    "reason": "index [logs-app-2026.10.20] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"
  },
  "status": 429
}

The 429 status means clients with retry logic will keep retrying, which is reasonable because the block is temporary.

The automatic release

Since 7.4, Elasticsearch removes the read_only_allow_delete block automatically once the node's disk usage falls below the high watermark. Before that, an operator had to clear it by hand, and many runbooks still contain the manual command. You only need it if you want to release the block before the next disk poll, or if the block was set by hand:

PUT logs-app-*/_settings
{
  "index.blocks.read_only_allow_delete": null
}

Releasing the block while the disk is still above flood stage is pointless; the next poll will set it again.

Max headroom: why percentages lie on large disks

On a 500 GB disk, 15% free is 75 GB, a sensible reserve. On a 20 TB disk, 15% free is 3 TB, a lot of disk sitting idle. Recent 8.x releases address this with max headroom settings, which cap the amount of free space each percentage watermark demands:

The effective rule is: a watermark is crossed when free space falls below the smaller of the percentage-derived free space and the max headroom. The default headrooms apply only when the corresponding watermark is left at its default percentage; if you set a watermark explicitly, set its headroom explicitly too if you want one.

Worked example: the same rules on two disk sizes

Node A, 1 TB disk (1,000 GB):

• Low: 15% free is 150 GB; max headroom is 200 GB. The smaller is 150 GB, so the low watermark trips at 850 GB used (85%).
• High: 10% free is 100 GB; headroom 150 GB. Smaller is 100 GB: trips at 900 GB used (90%).
• Flood: 5% free is 50 GB; headroom 100 GB. Smaller is 50 GB: trips at 950 GB used (95%).

On a 1 TB disk the headrooms never bind. The percentages rule.

Node B, 8 TB disk (8,000 GB):

• Low: 15% free is 1,200 GB; headroom 200 GB. Smaller is 200 GB: trips at 7,800 GB used (97.5%).
• High: 10% free is 800 GB; headroom 150 GB. Trips at 7,850 GB used (about 98.1%).
• Flood: 5% free is 400 GB; headroom 100 GB. Trips at 7,900 GB used (98.75%).

On the 8 TB node the cluster will fill to about 98% before it starts moving shards away. That uses expensive disk well, but it also means the gap between "start relocating" and "block writes" is only 50 GB. A single large shard or a burst of ingest can cross it before the relocation finishes. If your shards are large or your ingest is bursty, you may want explicit, larger headrooms. That is a judgment about your workload, not a default to copy.

Absolute values instead of percentages

The watermarks themselves can also be set as absolute free-space values, such as "500gb", meaning "trip when free space falls below 500 GB". You cannot mix types: either all three watermarks are percentages (or ratios) or all three are byte values. Mixing them is rejected.

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "400gb",
    "cluster.routing.allocation.disk.watermark.high": "250gb",
    "cluster.routing.allocation.disk.watermark.flood_stage": "100gb"
  }
}

Byte values are a reasonable choice in clusters where every data node has the same disk size and you want the thresholds to be obvious.

The frozen tier

Nodes in the frozen tier hold a shared cache for partially mounted searchable snapshots, not full shard copies, so they have their own flood stage setting, cluster.routing.allocation.disk.watermark.flood_stage.frozen (default 95%) with its own max headroom (default 20GB). See data tiers and ILM.

Seeing disk state the way the cluster sees it

The fastest overview:

GET _cat/allocation?v&s=disk.percent:desc

shards disk.indices disk.used disk.avail disk.total disk.percent host       ip         node
   412        1.6tb     1.8tb    196.4gb      2tb           90 10.0.1.13  10.0.1.13  es-hot-03
   398        1.4tb     1.5tb    478.2gb      2tb           76 10.0.1.11  10.0.1.11  es-hot-01
   401        1.4tb     1.5tb    490.7gb      2tb           75 10.0.1.12  10.0.1.12  es-hot-02

Two columns are worth comparing: disk.indices (bytes of shard data) and disk.used (bytes used on the filesystem). On es-hot-03, used exceeds indices by about 200 GB. That gap is not Elasticsearch shard data: look for logs, heap dumps, old snapshots copied locally, or another process writing to the same volume.

Recent 8.x releases also include the disk indicator in the health report, which states the problem and the affected indices in words:

GET _health_report/disk

The cluster settings currently in force, including defaults:

GET _cluster/settings?include_defaults=true&filter_path=*.cluster.routing.allocation.disk*

Always check this before assuming the defaults. A watermark changed months ago during an incident and never restored is a common finding.

Which indices are currently blocked:

GET _all/_settings/index.blocks.read_only_allow_delete?flat_settings=true

And whether any unassigned shard is waiting on disk:

GET _cluster/allocation/explain
{
  "index": "logs-app-2026.10.20",
  "shard": 0,
  "primary": false
}

A disk-blocked replica's explanation includes a disk_threshold decider entry saying the node is above the low watermark, with the actual percentage.

How shards move when a node crosses the high watermark

Knowing the relocation mechanics helps you predict how long recovery from a high-watermark state takes.

1. The allocator selects shards on the over-watermark node and target nodes that have room under their own low watermark and satisfy every other decider (awareness, filters, same-shard).
2. Recoveries are throttled. cluster.routing.allocation.node_concurrent_recoveries (default 2) limits concurrent incoming and outgoing recoveries per node. indices.recovery.max_bytes_per_sec limits recovery bandwidth, and the limit applies to each node separately; it defaults to 40mb, except on dedicated cold or frozen nodes, where the default scales with the node's total memory.
3. The target copies segment files, then replays operations that arrived during the copy. Only then is the source copy removed and its disk freed.

Worked example: draining 300 GB from a full node. Assume a node at 92% that needs to shed 300 GB to fall below the high watermark, with default recovery settings.

• At 40 MB per second, 300 GB is 300 x 1,024 / 40 = 7,680 seconds, a little over two hours, assuming the recovery throttle is the bottleneck.
• If ingest continues to add, say, 30 GB an hour to the shards on that node, the net drain is slower, and the node may cross flood stage first.

This is why "wait for it to rebalance" is not an incident plan for a fast-filling node. You either reduce writes to it, delete data, or temporarily raise the recovery rate if the network and disks can take it:

PUT _cluster/settings
{
  "persistent": {
    "indices.recovery.max_bytes_per_sec": "200mb"
  }
}

Set it back afterwards. Faster recovery costs disk and network bandwidth that ingest and search also need.

Planning headroom: the arithmetic

Watermarks protect a node. Headroom planning protects the cluster: it answers "if we lose a node, is there room to rebuild its shards elsewhere without hitting the high watermark?"

Worked example: a six-node hot tier

Assumptions: six hot data nodes, 2 TB disk each, one replica on every index, shards spread evenly.

1. Total disk: 6 x 2 TB = 12 TB.
2. Node loss: if one node fails, its 1/6 of the data must be rebuilt on the remaining five. Each survivor gains 1/5 of the lost node's data.
3. Target after loss: every survivor must stay below the low watermark (85%) so the rebuild can complete; below 85% the decider still accepts new copies.
4. Maximum steady-state usage per node, U: after loss, each survivor holds U + U/5 = 1.2U. We need 1.2U to be at most 85%, so U is at most 85 / 1.2, about 70.8%.

So on this cluster, a sustained steady-state disk usage above roughly 70% means that losing one node will push the survivors past the low watermark, leaving some replicas unassigned. The general formula for N equal nodes losing one is:

max steady-state usage = low watermark x (N - 1) / N

For N = 3 that is 85% x 2/3, about 56.7%. For N = 10 it is 76.5%. Small clusters need proportionally more free disk, which is one of the honest arguments for more, smaller data nodes over a few large ones.

Add merge and growth headroom

Merges create new segments before deleting the old ones, so a shard can temporarily need extra space; a force merge of a large shard down to one segment can transiently need roughly as much free space again as the shard's size. Add room for:

• The largest merge you will run. If you force merge 50 GB shards, keep at least that much free on the node, beyond the watermarks.
• Ingest growth until your next capacity change. If you add nodes quarterly, plan for a quarter's growth.
• Snapshot or restore activity, which can write large amounts of data in a short time during a restore.

A common rule of thumb for hot tiers is to plan for steady-state usage around 65 to 70%, then check that number against the node-loss formula above for your own N. For capacity sizing in logging clusters, see capacity planning for logging.

Failure modes and anti-patterns

The anti-pattern to avoid most is raising the watermarks during an incident and forgetting them. Moving flood stage to 98% buys a few gigabytes and removes most of the margin that protects the node from a genuinely full disk. A full disk is far worse than a blocked index: writes to the translog fail, shards can fail, and recovery becomes harder. If you must raise them to restore writes while you delete data, write down the change and restore it the same day:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": null,
    "cluster.routing.allocation.disk.watermark.high": null,
    "cluster.routing.allocation.disk.watermark.flood_stage": null
  }
}

Setting a value to null returns it to the default.

Another anti-pattern: deleting documents to free space. A delete by query marks documents as deleted; the space comes back only when merges rewrite the segments, which may take a long time and transiently needs more disk. Deleting whole indices frees space immediately. That is why time-based indices and data streams with ILM delete phases are the standard design for data with a retention period.

Runbook: a disk-full incident

1. Find the node or nodes.

   http GET _cat/allocation?v&s=disk.percent:desc

   Note which nodes are above 90% and 95% (or above the effective thresholds, if max headroom applies on large disks), and the gap between disk.indices and disk.used.

2. Confirm the settings in force with GET _cluster/settings?include_defaults=true, filtered to the disk settings. Do not assume defaults.

3. List the largest shards on the full node.

   http GET _cat/shards?v&h=index,shard,prirep,state,store,node&s=store:desc

   Filter the output for the node name. Large shards of old indices are deletion or migration candidates.

4. Free space by deleting whole indices, starting with data past its retention. For data streams, delete old backing indices or let ILM do it:

   http DELETE logs-app-2026.07.01,logs-app-2026.07.02,logs-app-2026.07.03

   Deletion is allowed under the read_only_allow_delete block. Name the indices explicitly: in 8.x, action.destructive_requires_name defaults to true, so a wildcard delete is refused, and that is a safeguard worth keeping during an incident.

5. If nothing can be deleted, stop or slow the writers sending data to the blocked indices, then move shards off the node, either by adding a data node or by excluding the full node so the allocator drains it:

   http PUT _cluster/settings { "persistent": { "cluster.routing.allocation.exclude._name": "es-hot-03" } }

   Only do this if the other nodes have room; otherwise you push them over their own watermarks.

6. Watch the block release. Once the node is below the high watermark, the block is removed on a later disk poll. Check:

   http GET _all/_settings/index.blocks.read_only_allow_delete?flat_settings=true

7. Restore anything you changed: watermark overrides, recovery rate, node exclusions (set them to null).

8. Write the headroom number down. Compute the steady-state ceiling with the node-loss formula and compare it with today's usage. If you are over it, the next incident is already scheduled.

Key takeaways

• Low (85%) stops new shard copies, high (90%) moves shards away, flood stage (95%) blocks writes to every index with a shard on the node.
• The flood-stage block is released automatically once usage falls below the high watermark; manual removal is rarely necessary.
• Max headroom caps the free space each percentage watermark demands (200, 150 and 100 GB by default). On large disks they bind and the cluster fills to around 98%, with a thin margin between relocating and blocking.
• Relocation is slow relative to fast ingest. A filling node needs deletion or reduced writes, not patience.
• Plan steady-state usage with the node-loss formula: low watermark x (N - 1) / N, then subtract merge and growth headroom.
• Never leave raised watermarks in place after an incident.

How Sparced helps

Sparced reads allocation, nodes and shards in one pass, so it shows each node's disk usage and shard count side by side. When the cluster is yellow or red, it calls _cluster/allocation/explain for each stuck shard, and when a disk watermark is the reason, the finding names the shard, the node and the decider's own explanation, with a fix command and a check filled in with those names. It does not delete data, change settings or move shards for you; it installs no agent and writes nothing to the cluster.