Elasticsearch Shard Sizing: A Capacity Model for Shard Counts

Most Elasticsearch capacity problems that look like heap pressure, slow searches or stuck recoveries are shard problems underneath: too many small shards, or a few shards so large that a node failure takes an hour to heal. The answer is not a magic number. It is a small capacity model: pick a target shard size from the workload, derive the primary count from the data volume, check the result against per-node limits, and let rollover hold the size steady over time. This article gives you that model, the arithmetic, and the commands to audit and correct an existing cluster.

Why shard count is a capacity decision

A shard is a complete Lucene index. Each one carries fixed costs regardless of how much data it holds:

• Segments and their in-memory structures. Every shard is made of immutable segments. Each segment has metadata, file handles and per-field structures that live partly on heap and partly in the filesystem cache.
• Cluster state. The elected master tracks every shard copy, its allocation and its state. Every index adds mappings and settings to the cluster state, which is published to every node on each change.
• Search fan-out. A search runs one task per shard copy it hits. A query against an index pattern that covers 3,000 shards schedules 3,000 shard-level tasks on the search thread pool, even if most of them return nothing.
• Recovery and relocation. Every shard is moved, recovered and replicated as a unit. The unit size determines how long the cluster stays yellow after a node restart.

So shard count trades off along two axes. Too many shards multiplies fixed overhead and fan-out. Too few, overly large shards make each recovery long, concentrate load on fewer nodes, and limit how much parallelism a single search can use.

What happens on a search

The coordinating node that receives a search resolves the target indices, picks one copy (primary or replica) of each shard, and sends a query-phase request to each. Each shard-level request occupies one thread from the node's search pool for its duration. The search pool size is int((allocated_processors * 3) / 2) + 1, with a queue of 1,000. On a 16-core data node that is 25 threads.

Do the arithmetic. A dashboard query over the last 30 days of a daily index with 5 primaries and 1 replica touches 30 x 5 = 150 shards. Spread over 6 data nodes, that is roughly 25 shard requests per node, which is the whole search pool of one 16-core node for one panel. Ten panels refreshing at once queue 250 shard requests per node. The same 30 days held in 30 shards (one primary per day, or larger rollover-based indices) is a fifth of that load.

This is why oversharding usually shows up first as search thread pool queueing and rejections, not as a heap alarm. See /guides/elasticsearch-thread-pools-rejections/ for reading those symptoms.

Elastic's guidance and the rules of thumb

Separate what Elastic documents from folklore.

Documented guidance (Elastic's shard sizing documentation for 8.x):

• Aim for shard sizes between 10 GB and 50 GB. Larger shards are possible but recover and rebalance more slowly.
• Aim for no more than about 200 million documents per shard.
• There is a hard Lucene limit of 2,147,483,519 documents per shard. You cannot exceed it, and indexing into a full shard fails.
• The default number of primary shards is 1 for a new index (it was 5 before 7.0), and the default number of replicas is 1.
• The per-node shard limit cluster.max_shards_per_node defaults to 1,000 for non-frozen data nodes, and cluster.max_shards_per_node.frozen defaults to 3,000 for nodes in the frozen tier.
• The built-in ILM policies used by data streams roll over at a maximum primary shard size of 50 GB or 30 days, whichever comes first.

Heap per shard, old versus new. Before 8.x, the widely quoted guidance was "no more than 20 shards per GB of heap". That figure was removed from Elastic's documentation in the 8.x line, after per-shard overhead was reduced. The documentation now frames data node heap cost in terms of mapped fields rather than shard count: there is heap overhead for each mapped field of each index a data node holds. Early 8.x documentation put a number on it, approximately 1 kB of heap per mapped field per index held by each data node; current documentation no longer quotes a figure, so treat 1 kB as a planning rule of thumb, not a documented limit. For master-eligible nodes the documented rule of thumb is fewer than 3,000 indices per GB of master heap. Treat the 20-per-GB figure as a 7.x-era rule of thumb that is conservative on 8.x and 9.x. It is still a useful smell test: if you are far past it, measure.

Rules of thumb (not documented limits):

• A common rule of thumb is 30 to 50 GB per shard for time-series and logging data and 10 to 30 GB per shard for search-heavy indices where latency matters more than storage density.
• A common rule of thumb is to keep the number of shards a single search touches in the low hundreds or fewer for interactive workloads.
• A common rule of thumb is that a shard should recover from a peer in minutes, not hours. At realistic recovery throughput, that caps practical shard size well below the point where Lucene itself struggles.

Field count is the hidden multiplier

If heap cost scales with fields per index per node, then a cluster of 2,000 indices each with 3,000 mapped fields carries 6 million field-index pairs. A data node holding a shard of every one of those indices needs on the order of 6 GB of heap just for that metadata under the 1 kB per field rule of thumb. Halving the index count halves that cost. This is the real reason daily indices for dozens of low-volume sources are expensive: not the data, the mapping replicated per index. Mapping hygiene matters as much as shard count; see /guides/elasticsearch-mapping-design/.

The per-node shard limit, precisely

cluster.max_shards_per_node is a dynamic cluster setting. The effective cluster-wide limit is:

max open shards (non-frozen) = cluster.max_shards_per_node x number of non-frozen data nodes

Details that matter in practice:

• It counts the shards of open indices, primaries and replicas both, including unassigned shards. Closed indices do not count.
• It is checked when an operation would add shards: creating an index, rollover, restoring a snapshot, opening a closed index, or increasing replicas. Existing shards are never removed because you crossed it.
• When the check fails, the operation is rejected with a validation error stating that the action would add N shards but the cluster already has X of a maximum Y shards open. On a data stream, this means rollover fails, the write index keeps growing past its target size, and ILM reports an error on the rollover step.
• It is a cluster-wide total derived from a per-node figure. It does not prevent one node from holding more than 1,000 shards if the allocator places them there. Use cluster.routing.allocation.total_shards_per_node (a hard per-node cap) or the index-level index.routing.allocation.total_shards_per_node if you need to bound a single node.

Check where you are:

GET _cluster/health?filter_path=number_of_data_nodes,active_shards,active_primary_shards,unassigned_shards

GET _cluster/settings?include_defaults=true&filter_path=**.max_shards_per_node*

A response like this:

{
  "number_of_data_nodes": 6,
  "active_primary_shards": 2410,
  "active_shards": 4820,
  "unassigned_shards": 0
}

with the default limit gives a ceiling of 6 x 1,000 = 6,000 open shards. You are at 4,820 (active plus 0 unassigned), or 80%. One more large daily index pattern with 5 primaries and 1 replica over 90 days of retention adds 900 shards and puts you at 5,720. The next onboarding will fail at index creation.

In recent 8.x releases the health report also has a shard capacity indicator, so GET _health_report will warn before you hit the wall.

Do not treat raising the limit as the fix. Raising cluster.max_shards_per_node is legitimate when you have measured that nodes are comfortable, but it is a guard rail. If you are hitting it, the usual root cause is daily indices with more primaries than their volume justifies.

Sizing time-series indices

For logs, metrics and events, the right tool is rollover: write through a data stream (or a rollover alias), and let ILM create a new backing index when the current one reaches a size or age. Rollover decouples shard size from the calendar. A source producing 5 GB a day and a source producing 500 GB a day can both end up with shards of about 40 GB.

The calculation has three inputs:

1. Primary data volume per day, measured on disk after indexing (not raw log bytes; the on-disk ratio depends on mappings, codec and _source).
2. Target primary shard size.
3. Retention.

Worked example: a high-volume log source

Assumptions:

• 600 GB per day of primary data on disk (measured as pri.store.size growth per day).
• Target primary shard size 40 GB.
• 1 replica.
• Retention 30 days, all on one tier for this example.
• 10 hot data nodes.

Step 1, primary shards per day: 600 / 40 = 15 primary shards' worth of data per day.

Step 2, choose primaries per backing index. You want each rollover to produce shards near 40 GB, and you want writes spread across nodes. With 10 hot nodes, a backing index with 5 primaries and 1 replica places 10 shard copies, one per node, which spreads write load evenly. Each backing index then holds 5 x 40 = 200 GB of primary data before rollover, so you roll over 600 / 200 = 3 times per day.

Step 3, total shards at steady state: 15 primaries per day x 30 days = 450 primaries, x 2 copies = 900 shards, or 90 per node. That is far below the limit and well within comfortable heap.

Step 4, sanity check the failure case. If one node fails, its 90 shards (90 x 40 GB = 3.6 TB) must be rebuilt elsewhere. At an effective recovery rate of, say, 200 MB/s per receiving node across 9 nodes (an assumption: it depends on indices.recovery.max_bytes_per_sec, disk and network), that is 3.6 TB / (9 x 0.2 GB/s) = 2,000 seconds, about 33 minutes of reduced redundancy. Note that 200 MB/s assumes you have raised the recovery throttle: indices.recovery.max_bytes_per_sec defaults to 40mb on hot nodes, and at 40 MB/s the same sum is 3.6 TB / (9 x 0.04 GB/s) = 10,000 seconds, nearly 3 hours. If that window is unacceptable, smaller shards will not help much; more nodes or faster recovery will.

The ILM rollover action for this:

PUT _ilm/policy/logs-highvol
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_size": "40gb",
            "max_age": "1d"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": { "delete": {} }
      }
    }
  }
}

and the shard count lives in the index template:

PUT _index_template/logs-highvol
{
  "index_patterns": ["logs-highvol-*"],
  "data_stream": {},
  "priority": 500,
  "template": {
    "settings": {
      "index.number_of_shards": 5,
      "index.number_of_replicas": 1,
      "index.lifecycle.name": "logs-highvol"
    }
  }
}

Note max_primary_shard_size is measured on the largest primary shard of the index, not the index total. That is the right control for shard sizing: it directly enforces the per-shard target.

Worked example: many low-volume sources

Assumptions: 120 application log streams, each producing between 50 MB and 2 GB per day, 1 replica, 90 days of retention, 6 data nodes.

The naive design is a daily index per application with 1 primary: 120 x 90 = 10,800 primaries, x 2 = 21,600 shards. With the default limit and 6 nodes the ceiling is 6,000. It fails long before reaching steady state, and most shards would be a few hundred MB, which is all overhead.

The rollover design with max_primary_shard_size: 40gb and max_age: 30d, 1 primary each: the largest stream (2 GB/day) would reach 60 GB in 30 days, which exceeds the 40 GB target, so it rolls over on size after about 20 days. Every other stream rolls over on age. Over 90 days each stream has about 3 to 5 backing indices. Call it 4 on average: 120 x 4 = 480 primaries, x 2 = 960 shards. Same data, 22 times fewer shards.

Better still, consolidate: if the 120 streams share a schema, write them into a handful of data streams and distinguish by a service.name field. The data stream naming scheme logs-<dataset>-<namespace> is built for exactly this grouping decision.

Why max_age still matters

Without an age condition, a stream that slows down can hold one backing index for months. Retention is applied per backing index, so a 90-day delete phase cannot remove data until the whole index is older than 90 days from rollover. A max_age bounds how long a single index stays open for writes and keeps deletion granular. In recent 8.x releases rollover also accepts minimum conditions (such as min_docs or min_primary_shard_size) to prevent rolling over tiny indices on age alone; use them when many streams are near-idle.

Sizing search indices

Search indices (product catalogs, documents, entities) are usually not time-series. They are updated in place, rebuilt occasionally, and queried with latency targets. The model is different:

1. Size the primary count for the data you will have at the end of the planning horizon, because index.number_of_shards is fixed at creation.
2. Size replicas for query throughput and availability, because replicas are dynamic and every replica can serve searches.

Worked example: a product catalog

Assumptions:

• Today: 180 GB of primary data, 120 million documents.
• Expected growth: to 400 GB and 260 million documents within two years.
• Latency-sensitive search; target shard size 20 GB.
• 6 data nodes.

Step 1, primaries for the horizon: 400 / 20 = 20 primaries. Check documents: 260M / 20 = 13M per shard, well under 200M.

Step 2, today's shard size with 20 primaries: 180 / 20 = 9 GB, slightly under the 10 GB lower bound but fine; it grows into the target.

Step 3, distribution: 20 primaries plus 1 replica is 40 copies on 6 nodes, 6 or 7 per node. A slightly better choice is 18 or 24 primaries, which divide evenly into 6 nodes (with 1 replica, 36 or 48 copies). Even distribution means no node does more than its share of every query. Choose 24 primaries: 400 / 24 = about 17 GB at the horizon.

Step 4, replicas for throughput: measure. If one copy of the index sustains 400 queries per second at your latency target on 6 nodes, a second replica does not double throughput automatically (the nodes are shared), but adding nodes and replicas together scales it. Replicas are cheap to change; primaries are not.

Step 5, leave room to split. Since 7.0, an index's default index.number_of_routing_shards allows the split API to multiply the primary count by powers of 2, up to 1,024 shards. Picking 24 now means you can split to 48 later without reindexing if the growth forecast is wrong.

When a single shard is better

For a small index (a few GB) that is searched heavily, 1 primary with several replicas often beats several primaries. Each search touches one shard instead of many, so per-query cost drops, and replicas spread the query load. The rule is: shard for size, replicate for throughput.

Decision table

Auditing an existing cluster

Start with the shards themselves, sorted by size:

GET _cat/shards?v&h=index,shard,prirep,state,docs,store,node&s=store:desc

Then the small end, which is where oversharding lives:

GET _cat/shards?v&h=index,shard,prirep,store&s=store:asc&bytes=mb

Index level, with primary size and shard count:

GET _cat/indices?v&h=index,health,pri,rep,docs.count,pri.store.size,store.size&s=pri.store.size:desc&bytes=gb

Per node, shard count and disk:

GET _cat/allocation?v

Questions to ask of the output:

• What fraction of shards are under 1 GB? If it is most of them, you are oversharded. A shard holding 50 MB costs nearly the same overhead as one holding 5 GB.
• What is the largest shard? Anything over 100 GB will be slow to recover and relocate. Anything near the document limit is an incident waiting to happen.
• Is shard count even across nodes? The allocator balances counts and, in recent 8.x releases, also disk usage and write load. Big skews usually trace back to allocation filters or tier preferences.
• How many shards does your heaviest dashboard touch? Use the _shards.total field in a search response to find out.

GET logs-*/_search?size=0
{
  "query": { "range": { "@timestamp": { "gte": "now-30d" } } }
}

{
  "took": 412,
  "timed_out": false,
  "_shards": { "total": 1350, "successful": 1350, "skipped": 1180, "failed": 0 }
}

skipped counts shards the coordinating node excluded in the pre-filter phase because their timestamp range could not match. A high skipped is good: it means those shards were cheap. A high total with low skipped is the expensive case.

Fixing oversharding and undersharding

Shard count is fixed at index creation, but there are four ways out.

1. Fix the template for new indices

The cheapest fix: change index.number_of_shards in the index template and the rollover conditions in the ILM policy. New backing indices pick it up at the next rollover. Old ones age out under retention. For a 30-day retention, the problem is gone in a month with no reindexing.

To apply immediately on a data stream:

POST logs-highvol-default/_rollover

2. Shrink existing indices

The shrink API reduces primaries to a factor of the current count (for example 12 to 6, 4, 3, 2 or 1). Requirements: the index must be read-only, and a copy of every shard must be on one node.

PUT logs-app-2026.09.01/_settings
{
  "index.routing.allocation.require._name": "data-node-3",
  "index.blocks.write": true
}

POST logs-app-2026.09.01/_shrink/logs-app-2026.09.01-shrunk
{
  "settings": {
    "index.number_of_shards": 1,
    "index.number_of_replicas": 1,
    "index.routing.allocation.require._name": null,
    "index.blocks.write": null
  }
}

ILM's shrink action in the warm phase does all of this for you, and is the right way for time-series data. Shrink uses hard links where the filesystem allows, so it is fast on the same node.

3. Split an index that is too large

The split API multiplies the primary count. The target must be a multiple of the source, within what index.number_of_routing_shards allows. The source must be write-blocked.

PUT catalog-v3/_settings
{ "index.blocks.write": true }

POST catalog-v3/_split/catalog-v4
{
  "settings": { "index.number_of_shards": 48 }
}

Then move the alias from catalog-v3 to catalog-v4 and remove the write block.

4. Reindex

When the change is not a factor or multiple, or when you are also changing mappings, reindex into a new index and swap an alias. This is the most flexible and most expensive option. See /guides/elasticsearch-zero-downtime-reindex/.

Merging many daily indices

Consolidating 90 daily indices of one source into monthly indices is a reindex job: create the monthly target with the right shard count, reindex logs-app-2026.09.* into it, verify counts, point the alias, delete the dailies. Do it month by month and throttle with requests_per_second so ingestion is not starved.

Failure modes and anti-patterns

Daily indices with the default 5 primaries carried forward from old templates. Symptom: thousands of shards under 1 GB, rising cluster state size, slow master operations, search queueing on dashboard refresh. Confirm: _cat/shards sorted by size ascending; count shards below 1 GB. Fix: template change to 1 primary and size-based rollover; shrink or let old indices age out.

Rollover not happening, shards growing without bound. Symptom: one backing index at 300 GB per shard, slow recoveries, ILM errors. Confirm: GET <data-stream>/_ilm/explain shows the rollover step in error, often from the shard limit or a missing alias on a non-data-stream setup. Fix: resolve the error, then POST <target>/_ilm/retry (or roll over manually). See /guides/elasticsearch-ilm-troubleshooting/.

Hitting cluster.max_shards_per_node during an incident. Symptom: index creation fails with a "maximum shards open" validation error; data streams stop rolling over. Confirm: compare active_shards plus unassigned_shards with limit x non-frozen data nodes. Fix in the moment: close or delete indices you do not need, or reduce replicas on non-critical old indices. Raise the limit only as a temporary measure and record why.

Primary count not a divisor or multiple of the node count. Symptom: one or two nodes consistently hotter on indexing. Confirm: _cat/shards for the write index; count copies per node. Fix: choose a primary count that divides evenly across hot nodes, or set index.routing.allocation.total_shards_per_node on the write index to force spread. Use that setting carefully: if a node fails and the cap leaves no legal home for a shard, it stays unassigned.

Shards so large that recovery dominates. Symptom: a single node restart leaves the cluster yellow for hours. Confirm: GET _cat/recovery?v&active_only=true shows long-running recoveries on a few shards of hundreds of GB. Fix: smaller rollover target; more nodes; check recovery throttles.

Too many replicas on large indices. Symptom: disk fills, high watermark triggers relocations, indexing slows because each document is written several times. Fix: replicas are for availability and read throughput. One is usually enough for logs. See /guides/elasticsearch-disk-watermarks/.

Checklist: sizing a new index or data stream

1. Measure on-disk primary bytes per day (index a representative sample for a day, read pri.store.size).
2. Choose a target primary shard size: 30 to 50 GB for time-series, 10 to 30 GB for latency-sensitive search (rule of thumb within Elastic's 10 to 50 GB guidance).
3. For time-series: set primaries per backing index to the number of hot data nodes or a divisor of it; set rollover max_primary_shard_size to the target and a max_age that bounds retention granularity.
4. For search: primaries = horizon volume / target size, rounded to a multiple of the data node count; confirm documents per shard under 200M.
5. Compute steady-state total shards: primaries per day x retention days x (1 + replicas). Compare with cluster.max_shards_per_node x non-frozen data nodes. Keep headroom of at least a quarter for growth and incidents (rule of thumb).
6. Estimate field-count heap: indices per node x mapped fields x 1 kB (rule of thumb), and compare with data node heap.
7. Estimate recovery time for one node loss: shards per node x shard size / aggregate recovery throughput.
8. Put shard count in the index template, size control in ILM, and verify with _cat/shards after the first few rollovers.

Key takeaways

• Shard count is a capacity decision with fixed per-shard cost: cluster state, heap for mapped fields, and search fan-out on the search thread pool.
• Elastic documents 10 to 50 GB per shard and about 200 million documents per shard; Lucene's hard limit is 2,147,483,519 documents.
• cluster.max_shards_per_node defaults to 1,000 (3,000 for frozen) and is enforced as a cluster total when shards are added, which is how rollover silently stalls.
• For time-series, size with rollover on max_primary_shard_size plus max_age, not with daily indices.
• For search, fix primaries for the planning horizon and use replicas, which are dynamic, for throughput.
• Shrink, split and reindex correct existing indices; changing the template fixes everything new.

How Sparced helps

Sparced reads cluster health, nodes, shards, allocation and ILM state in one pass, so the shard count per node, the size distribution of shards and any index whose ILM rollover step is in error appear together as findings, each listing the actual indices and nodes involved. When the cluster is red or yellow, it calls _cluster/allocation/explain for each stuck shard and shows the cluster's own reason next to the evidence. Each finding comes with a fix command with your index and node names filled in, and a check to confirm it worked. It writes nothing to the cluster, so the shrink, template or rollover change is yours to apply; Sparced does not choose a shard count for you.