Elasticsearch Capacity Planning for Logging: A Worked Example

Capacity planning for a logging cluster is mostly arithmetic, and the arithmetic is not hard. What goes wrong is the inputs: a raw-to-indexed ratio guessed instead of measured, replicas forgotten on one tier, disk planned to 100% instead of to the watermarks, a node failure that pushes every survivor over the high watermark, or a frozen tier that runs out of shards long before it runs out of disk. The method in this article is to measure four numbers from your own data, then work tier by tier from ingest volume to stored bytes to usable disk to node count, and check the result against the second and third constraints (ingest rate and shard count) before you buy anything. Every step is shown, with the assumptions stated so you can replace them.

The model: what turns raw logs into disk

A log event goes through several transformations between the source and the disk, and each one changes its size.

1. Raw bytes. What the shippers read: the log line or JSON object.
2. Parsed document. After parsing, the document has fields extracted. It may be larger (added metadata, ECS fields, geo data) or smaller (dropped fields).
3. Indexed bytes. Elasticsearch stores several structures per document. The stored _source is compressed. The inverted index for text and keyword fields, doc values for aggregatable fields, and points for numbers and dates are each separate on-disk structures in Lucene segments. Their total depends heavily on the mapping.
4. Replicas. Each replica is a full copy of the primary's indexed bytes on another node.
5. Merges and overhead. Lucene writes immutable segments and merges them in the background. During a merge the old and new segments coexist on disk, so a shard temporarily needs more space than its steady state. Translog files add a little more.

The ratio that matters for planning is indexed primary size divided by raw size, measured on your data with your mappings. A common rule of thumb for typical log data with default dynamic mappings and _source kept is somewhere around 1.0 to 1.5 times raw. Careful mappings, best_compression, and newer storage modes can bring it well below 1. Do not plan from the rule of thumb; measure it.

Measuring the ratio

Index one representative day (or a few hours at a known rate) into a test index with the production template, force merge it to approximate steady state, and compare with the raw bytes you sent:

POST logs-sample-test/_forcemerge?max_num_segments=1

GET _cat/indices/logs-sample-test?v&h=index,docs.count,pri.store.size,store.size&bytes=gb

index             docs.count pri.store.size store.size
logs-sample-test    48211033          41.7       83.4

If those 48 million events were 38.0 GB raw, the ratio is 41.7 / 38.0 = 1.10. pri.store.size excludes replicas; store.size includes them. Use the primary figure and apply replicas explicitly per tier, because the replica count differs by tier.

Repeat for each major data source if their shapes differ much. Firewall logs with many short fields and application logs with long messages compress very differently.

Levers that change the ratio

These are covered in depth in /guides/elasticsearch-storage-cost-optimization/. For planning, the point is that the ratio is a design choice, and a 30% better ratio is 30% fewer hot nodes.

The four inputs

Before any arithmetic, pin down these four numbers. Each one wrong by a factor of two makes the plan wrong by a factor of two.

1. Raw ingest per day, and its peak rate. Measure at the shippers or the buffer, not from a vendor estimate. Note the peak hour as a multiple of the average hour.
2. Indexed-to-raw ratio, measured as above.
3. Retention per tier: how many days must be fast to search (hot), how many must be searchable at moderate speed (warm or cold), and how many must be searchable at all (frozen) or merely kept (snapshots only).
4. Replica policy per tier: typically one replica on hot and warm, zero on cold and frozen when they are backed by searchable snapshots, because the snapshot repository already holds a durable copy.

Then three planning constants:

• Target disk utilization. The low watermark default is 85%, high 90%, flood stage 95%. Planning to a steady state well below the low watermark leaves room for merges, relocations and growth. A common rule of thumb is to plan to about 75% so that the cluster can lose a node and stay below the low watermark; the worked example checks this explicitly.
• Usable disk per node for each tier, after RAID and filesystem overhead.
• Shard size target. Elastic's guidance for logging data is shards between roughly 10 GB and 50 GB and fewer than 200 million documents each. ILM rollover on max_primary_shard_size enforces this. See /guides/elasticsearch-shard-sizing/.

The worked example inputs

The example cluster:

• Raw ingest: 2.0 TB per day. Peak hour 3 times the average hour.
• Measured indexed-to-raw ratio: 1.10.
• Retention: 7 days hot, then warm until day 30, then frozen until day 365. Snapshots retained for the full year (frozen depends on them).
• Replicas: 1 on hot, 1 on warm, 0 on frozen.
• Hot nodes: 4.0 TB usable fast local disk, 64 GB RAM.
• Warm nodes: 16 TB usable disk, 64 GB RAM.
• Frozen nodes: 8 TB local disk for the shared cache, 64 GB RAM.
• Target disk utilization: 75% on hot and warm.

All of these are assumptions for illustration. Substitute your own.

Hot tier

Storage

1. Indexed primary per day: 2.0 TB x 1.10 = 2.2 TB.
2. With one replica: 2.2 x 2 = 4.4 TB per day.
3. Seven days on hot: 4.4 x 7 = 30.8 TB.
4. Add one more day of buffer, because rollover and the ILM move to warm are not instantaneous and a day's data is still being written: 4.4 x 8 = 35.2 TB. (A rule of thumb; the buffer should cover however long your phase transition takes in practice.)
5. At 75% target utilization: 35.2 / 0.75 = 46.9 TB of raw disk.
6. At 4.0 TB per node: 46.9 / 4.0 = 11.7, so 12 hot nodes.

Node-loss check

With 12 nodes holding 35.2 TB, each holds 35.2 / 12 = 2.93 TB, or 2.93 / 4.0 = 73% of its disk.

If one node fails and its replicas are rebuilt on the others, 11 nodes hold 35.2 TB: 35.2 / 11 = 3.2 TB each, 3.2 / 4.0 = 80%. That is below the 85% low watermark, so recovery completes without hitting watermarks. With 11 nodes instead of 12, the same check gives 35.2 / 10 = 3.52 TB, 88% after a failure, above the low watermark: replicas of the lost node could not all be allocated. The extra node is the cost of surviving a failure cleanly.

Ingest rate check

Disk is not the only constraint on hot. The hot tier must also absorb the peak indexing rate.

1. Average raw rate: 2.0 TB / 86,400 s = 23.1 MB/s.
2. Peak raw rate: 23.1 x 3 = 69.4 MB/s.
3. Every byte is indexed on the primary and again on the replica, so indexing work at peak is about 69.4 x 2 = 138.9 MB/s of raw-equivalent input across the tier.
4. Suppose a load test shows one hot node sustains 20 MB/s of raw-equivalent indexing (primary plus replica work) with acceptable search latency. That number depends on your hardware, mappings and pipelines; measure it with the production template.
5. Nodes for peak: 138.9 / 20 = 6.9, so 7 nodes.

Disk needs 12 nodes and ingest needs 7, so disk governs, and the hot tier has ingest headroom. If the ingest check had given a larger number, ingest would govern, and the answer is more hot nodes with smaller disks, not bigger disks. If a buffer such as Kafka absorbs peaks, you can size ingest for average plus catch-up instead; see /guides/logging-architecture-kafka-elasticsearch/.

Memory

Each hot node has 64 GB RAM. Elastic's guidance is a heap of no more than 50% of RAM and below the compressed-oops threshold, which is near 32 GB and varies by JVM (26 GB is safe on most systems; up to about 30 GB where the startup log confirms compressed oops), so about 30 GB heap, leaving the rest for the filesystem cache, which Lucene relies on for fast search. Recent versions size the heap automatically from available memory and node roles if you do not set it. See /guides/elasticsearch-jvm-heap-circuit-breakers/.

Warm tier

1. Days on warm: day 8 to day 30, so 23 days.
2. Data: 4.4 TB per day x 23 = 101.2 TB.
3. At 75%: 101.2 / 0.75 = 134.9 TB raw disk.
4. At 16 TB per node: 134.9 / 16 = 8.4, so 9 warm nodes.

Node-loss check: 9 nodes hold 101.2 TB, 11.2 TB each, 70% of 16 TB. After losing one, 8 nodes hold 12.65 TB each, 79%. Below 85%, so recovery completes.

A note on warm: force merging indices to one segment per shard when they enter warm can reduce their size slightly and makes them cheaper to search, at the cost of I/O during the merge. The sizing above does not assume a saving; if your measurements show one, apply it.

Recovery time matters more on warm than on hot. Each warm node holds 11 TB; rebuilding that over the network after a node failure takes hours. At an effective 200 MB/s of recovery throughput into the surviving nodes (an assumption; recovery is throttled by indices.recovery.max_bytes_per_sec and by disk), 11.2 TB takes 11,200,000 MB / 200 MB/s = 56,000 s, about 15.6 hours. During that window the affected shards have no replica. That is a reason to prefer more, smaller warm nodes if the window is too long for your risk tolerance.

Cold tier, or not

A cold tier holds fully mounted searchable snapshots: the data is restored to local disk from the snapshot repository, with no replicas, because a lost copy is restored from the repository again. It roughly halves storage compared with warm (no replica) and keeps search fast.

In this example, the requirement is "searchable, rarely", so frozen is the right choice for days 31 to 365. See /guides/elasticsearch-data-tiers-ilm/ for how the tiers and ILM phases fit together.

Frozen tier

Repository storage

Frozen indices are partially mounted searchable snapshots. The full data lives in the snapshot repository (object storage); the frozen nodes hold only a local cache.

1. Days on frozen: day 31 to day 365, so 335 days.
2. Primary data per day: 2.2 TB (no replicas in a snapshot).
3. Repository storage for frozen: 2.2 x 335 = 737 TB.

Snapshots are incremental at the segment level, and the hot and warm data is snapshotted too, so total repository storage is somewhat higher than this. Size and price the object store accordingly.

Cache and nodes

On a node with only the frozen data role, the shared cache (xpack.searchable.snapshot.shared_cache.size) defaults to 90% of the node's disk, capped by a headroom setting, xpack.searchable.snapshot.shared_cache.size.max_headroom, which defaults to 100 GB in that case. On a large disk the headroom wins: an 8 TB (8,000 GB) disk at 90% would leave 800 GB free, so the cache is sized at 8,000 - 100 = 7,900 GB instead. The cache holds the parts of the data that have been searched recently.

How big the cache must be depends on what users search, not on how much data exists. This is the least predictable number in the plan. For the example, assume searches over the frozen tier mostly touch recent months and that a cache of about 5% of frozen data keeps typical searches acceptable. That assumption must be tested with your query patterns.

1. Cache needed: 737 TB x 0.05 = 36.9 TB.
2. Cache per node: 8 TB minus the 100 GB headroom = 7.9 TB.
3. Nodes for cache: 36.9 / 7.9 = 4.7, so 5 nodes. Hold that number until the shard count check below.

Shard count check

Frozen is where shard counts bite.

1. Primary data per day 2.2 TB, and the rollover target is 50 GB per primary shard, so at least 2,200 / 50 = 44 primary shards per day across all data streams. Real counts are higher, because many small data streams roll over on age before reaching 50 GB.
2. Frozen holds 335 days: 44 x 335 = 14,740 shards (no replicas).
3. Frozen nodes have their own shard limit, cluster.max_shards_per_node.frozen, default 3,000 per frozen node.
4. 5 frozen nodes x 3,000 = 15,000. 14,740 is 98% of that limit, with no allowance for small data streams, so 5 nodes is not a plan.
5. 6 frozen nodes x 3,000 = 18,000. 14,740 fits at 82% of the limit. On this tier the shard limit, not the cache, sets the node count: 6 frozen nodes.

If your real per-day shard count is higher, frozen will hit the shard limit before it runs out of cache, and index mounting will fail. Options: fewer, larger data streams; rollover on larger sizes where search allows; more frozen nodes; or a shrink step before frozen. Plan this before the first year of data accumulates.

For hot and warm, check cluster.max_shards_per_node (default 1,000 per non-frozen data node): hot holds 44 x 2 x 8 = 704 shards across 12 nodes and warm 44 x 2 x 23 = 2,024 across 9, both far inside the limit.

Masters, coordinating nodes and the full bill

The data tiers are most of the cost but not all of it.

• Masters: 3 dedicated master-eligible nodes. They need modest CPU, memory and disk, but they must be stable. Cluster state grows with indices, shards and mapped fields; 15,000 frozen shards plus hot and warm is a substantial cluster state, so do not under-size them. See /guides/elasticsearch-cluster-topology/.
• Coordinating or ingest nodes: only if parsing is heavy or many clients connect directly. Size from measured pipeline cost; see /guides/ingest-pipelines-vs-logstash/.
• Kibana: at least 2 instances.
• Monitoring cluster: small, separate. See /guides/monitoring-elasticsearch-production/.

The example's bill of materials:

Sensitivity: which input moves the answer most

Rerun the hot-tier calculation with each input changed to see what you are really betting on:

Two observations. The ratio is as powerful a lever as retention, and it is under your control through mapping work. And cutting hot retention quickly makes ingest the governing constraint instead of disk, which is why both checks must always be run.

Failure modes and anti-patterns

Planning from raw bytes. Symptom: the cluster fills months earlier than planned. Cause: indexed size is larger than raw. Fix: measure the ratio on your data.

Forgetting the replica on one tier. Symptom: warm fills at half the planned retention. Fix: write replicas into the calculation per tier.

Planning to 100% disk. Symptom: low watermark reached in steady state, so replicas of new indices stay unassigned; once nodes pass the high watermark even new primaries cannot be placed and shards start relocating away, so a rollover can leave an unassigned write index. Fix: plan to about 75% and run the node-loss check. See /guides/elasticsearch-disk-watermarks/.

No node-loss headroom. Symptom: after one node fails, the cluster stays yellow because replicas cannot be placed below the watermarks. Confirm with _cluster/allocation/explain showing the disk threshold decider. Fix: one more node per tier, or lower utilization.

Disk sized, ingest not. Symptom: write rejections at peak on a cluster with plenty of disk. Fix: size hot for peak indexing too, or buffer peaks.

Frozen shard limit. Symptom: after months of accumulation, mounting new frozen indices fails. Confirm with the shard count per frozen node against cluster.max_shards_per_node.frozen. Fix: larger shards, fewer data streams, more frozen nodes.

Under-measured frozen cache. Symptom: frozen searches take minutes. Cause: the cache is far smaller than the working set. Fix: measure which time ranges users actually search and resize.

Growth not in the plan. Log volume grows with the business. Re-run the plan quarterly with measured inputs, and add a growth factor for the procurement lead time.

Runbook: building or revalidating a capacity plan

1. Measure raw ingest per day and the peak-hour multiple at the shippers or buffer, over at least two weeks.
2. Measure the indexed-to-raw ratio per major data source with the production template and a force merge.
3. Confirm retention per tier and replica policy per tier with the data owners, in writing.
4. Compute hot storage: daily indexed primary x (1 + replicas) x (hot days + buffer days) / target utilization. Divide by usable disk per node.
5. Run the hot node-loss check: data / (nodes - 1) / disk per node must stay below the low watermark (85% by default).
6. Run the hot ingest check from a measured per-node indexing rate at peak. Take the larger node count.
7. Repeat storage and node-loss checks for warm or cold.
8. Compute repository storage for frozen, then frozen nodes from a stated cache assumption.
9. Compute shard counts per tier and compare with cluster.max_shards_per_node and cluster.max_shards_per_node.frozen.
10. On a live cluster, compare the plan with reality: GET _cat/allocation?v for disk per node, GET _cat/indices?v&h=index,pri.store.size,store.size&s=index for actual sizes, GET _cluster/health for shard totals. Update the inputs and rerun.

Key takeaways

• Measure the indexed-to-raw ratio on your own data; it moves the answer as much as retention does.
• Apply replicas per tier: usually one on hot and warm, zero on cold and frozen backed by snapshots.
• Plan disk to about 75% and prove the tier stays under the 85% low watermark after losing a node.
• Check ingest rate on hot and shard counts on every tier; disk is only one of three constraints.
• Frozen capacity is set by the cache your searches need and by the frozen shard limit, not by data volume alone.
• Rerun the plan with measured inputs every quarter.

How Sparced helps

Sparced reads the live inputs to step 10 in one pass: disk use per node against the watermarks, shard counts and their distribution across nodes, allocation state, ILM status per index and snapshot status. If replicas cannot be placed after a node loss, the cluster goes yellow, and Sparced calls _cluster/allocation/explain for each stuck shard, so a disk threshold decision that shows your plan has no headroom appears as a finding with the shards and nodes named. It also flags heap above 85% on data nodes. It does not produce a capacity plan or forecast growth; the arithmetic in this article is still yours, and Sparced tells you where the running cluster already disagrees with it.