Elasticsearch JVM Heap, G1GC and Circuit Breakers Explained

Most Elasticsearch memory incidents look the same from the outside: a node's heap climbs past 85 or 90 percent, searches start failing with circuit_breaking_exception and HTTP 429, garbage collection pauses get longer, and eventually a node drops out of the cluster. The answer is almost never "give it more heap". It is to size the heap correctly once (no more than half the machine's RAM, below the compressed-oops threshold), to read heap usage after garbage collection rather than the raw sawtooth, and to treat every breaker trip as evidence pointing at a specific request, field or pattern. This article gives you the model behind the heap, the breakers that guard it, the commands that show you what is consuming it, and a runbook for the day a node is in trouble.

The mental model: two kinds of memory on every node

An Elasticsearch node uses memory in two very different places, and confusing them is the root of most bad sizing decisions.

The JVM heap holds Java objects: the cluster state, in-flight request and response buffers, aggregation buckets, fielddata, the node query cache, the shard request cache, the indexing buffer, and per-shard and per-segment bookkeeping. Everything the circuit breakers account for lives here.

The off-heap memory, mostly the operating system's filesystem cache, holds Lucene's index files. Lucene memory-maps segment files and relies on the kernel to keep hot parts of them in RAM. Doc values, postings lists, stored fields, points and vectors are all read through the page cache. Elasticsearch does not manage this memory; the kernel does, using whatever RAM is not claimed by the heap and other processes.

This is why the heap sizing guidance is a ceiling, not a target. Every gigabyte you give to the heap is a gigabyte the filesystem cache cannot use. A node with a 60 GB heap on a 64 GB machine will have a large heap and dreadful search performance, because every query reads from disk.

A third, smaller category is direct (off-heap) memory used by the networking layer and some native components such as machine learning processes. It matters for container memory limits but is rarely the cause of a breaker trip.

Sizing the heap: the documented rules

Elastic's documented guidance comes down to three rules:

1. Set the minimum and maximum heap to the same value (-Xms equal to -Xmx), so the JVM never resizes the heap at runtime.
2. Use no more than 50% of the machine's (or container's) RAM for the heap. The rest goes to the filesystem cache and off-heap needs.
3. Stay at or below the compressed ordinary object pointers (compressed oops) threshold, which is near 32 GB. Elastic's documentation says the exact threshold varies, but 26 GB is safe on most systems and it can be as large as 30 GB on some.

Recent versions of Elasticsearch size the heap automatically by default, based on the node's roles and the total memory available to it. For most deployments that automatic sizing is the right answer, and you should only override it when you have a specific reason.

When you do override it, do not edit the bundled jvm.options file. Put a file in the jvm.options.d directory instead, for example config/jvm.options.d/heap.options:

-Xms26g
-Xmx26g

Why the compressed-oops threshold matters

Below the threshold, the JVM stores object references as 32-bit offsets and scales them, so a 32-bit pointer can address roughly 32 GB of heap. Above it, every reference becomes a full 64-bit pointer. The heap fills faster for the same data, cache efficiency drops, and a 34 GB heap can hold less useful data than a 30 GB one. Crossing the threshold is not a gentle slope; it is a step down.

You can confirm what the JVM chose from the startup log. Elasticsearch prints a line like this when the node starts:

heap size [26gb], compressed ordinary object pointers [true]

If that says false, your heap is too large for compressed oops on that JVM. Reduce it.

Worked example: sizing a data node

Assumptions: a hot data node with 64 GB of RAM, running only Elasticsearch, no machine learning.

• 50% of 64 GB is 32 GB. That is the upper bound from rule 2.
• 32 GB is at or above the compressed-oops threshold, so rule 3 caps it lower.
• Choose 26 GB, the value Elastic documents as safe on most systems, or up to 30 GB if the startup log confirms compressed oops at that size. Check the startup log for compressed ordinary object pointers [true].
• At 26 GB of heap, roughly 38 GB is left for the filesystem cache, the OS and direct memory.

On a 128 GB machine the arithmetic changes only on the cache side: the heap stays at 26 to 30 GB (rule 3 binds before rule 2), and the extra RAM goes to the filesystem cache. A common rule of thumb is that past about 64 GB of RAM per node, extra memory helps search through caching rather than through a bigger heap. If you need more heap capacity for the cluster, you add nodes, not heap.

For containers, the 50% rule applies to the container's memory limit, not the host. If a pod has a 16 GB limit, an 8 GB heap is the ceiling.

Swapping and memory locking

A JVM whose heap is partly swapped out will stall on garbage collection while pages come back from disk. Elastic's guidance is to disable swap on Elasticsearch hosts, or, where you cannot, to lock the heap in memory with this line in elasticsearch.yml:

bootstrap.memory_lock: true

This only works if the process is allowed to lock memory (the memlock ulimit, or the equivalent systemd setting). Verify it took effect:

GET _nodes?filter_path=**.mlockall

Every node should report "mlockall": true if you configured it.

Garbage collection: how G1GC behaves on Elasticsearch

The JDK bundled with Elasticsearch 8.x and 9.x uses the G1 garbage collector, and Elasticsearch's default JVM options are tuned for it. You should rarely need to change GC settings, and changing them without measurement is a common way to make things worse.

G1 divides the heap into equal-sized regions. Young regions receive newly allocated objects. Objects that survive several young collections are copied into old regions. Three kinds of activity matter operationally:

• Young collections are frequent and short. They are normal. A busy node may run several per second, each taking milliseconds to tens of milliseconds.
• Concurrent marking cycles run when old-generation occupancy crosses an initiating threshold. They work alongside the application and are followed by mixed collections that reclaim old regions.
• Full GCs happen when G1 cannot keep up: the old generation fills before concurrent marking and mixed collections free enough space. A full GC is stop-the-world and can last seconds on a large heap. A node in a full GC does not answer the master, and if the pause outlasts the fault-detection timeouts the master removes it from the cluster.

The sawtooth and the real signal

Heap usage on a healthy node is a sawtooth. It rises as requests allocate objects and falls after each collection. A raw heap.percent of 75% tells you very little: it may be seconds away from dropping to 30%.

The number that matters is how full the old generation is after collections, sometimes called the heap floor. If the floor keeps rising across hours, something is retaining memory: a growing cache, fielddata, too many shards, or a large cluster state. If the floor is stable but peaks spike, individual requests are allocating heavily.

A common rule of thumb is that sustained old-generation occupancy above about 75% deserves investigation and above about 85% is pressure that will soon show up as breaker trips and long pauses. Elastic Cloud's "JVM memory pressure" metric is built on the same idea: it is based on old-generation usage rather than total heap.

Read the GC numbers directly:

GET _nodes/stats/jvm?filter_path=nodes.*.name,nodes.*.jvm.mem.heap_used_percent,nodes.*.jvm.mem.pools.old,nodes.*.jvm.gc.collectors

The fields that matter:

{
  "nodes": {
    "a1b2": {
      "name": "es-hot-03",
      "jvm": {
        "mem": {
          "heap_used_percent": 81,
          "pools": {
            "old": {
              "used_in_bytes": 19864223744,
              "max_in_bytes": 27917287424
            }
          }
        },
        "gc": {
          "collectors": {
            "young": { "collection_count": 912345, "collection_time_in_millis": 8123456 },
            "old":   { "collection_count": 41,     "collection_time_in_millis": 96210 }
          }
        }
      }
    }
  }
}

collection_count and collection_time_in_millis are cumulative since node start. A single sample says nothing; take two samples a minute apart and compute the difference. As a rule of thumb, if young collection time is growing by more than a few seconds per minute, or old collections are counting up at all on a node that normally has none, the node is under pressure.

GC logs are written to the logs directory by default (gc.log and rotated files). For a pause investigation they are the authoritative record of every collection and its duration.

Circuit breakers: what they are and what they are not

A circuit breaker is an accounting mechanism. Before Elasticsearch performs an operation that will allocate a significant amount of memory, it estimates the bytes, adds them to the relevant breaker, and checks the total against a limit. If the total would exceed the limit, the operation is refused with a circuit_breaking_exception instead of being allowed to push the node into an OutOfMemoryError.

Two consequences follow:

• A breaker trip is a symptom of memory pressure, not its cause. The breaker did its job: it failed one request so the node would survive.
• Breakers are estimates. Most child breakers track memory that Elasticsearch explicitly reserves. The parent breaker, by default, looks at actual heap usage, which is why it is the one that trips most often in practice.

The breakers and their defaults

Each child breaker also has an overhead constant, a multiplier applied to its estimates. Fielddata uses 1.03 and in-flight requests uses 2; leave these alone.

There are additional breakers for specific features (for example for EQL sequences and for machine learning model inference), and a separate script compilation limit that rate-limits how often new scripts can be compiled. The five above are the ones you will see in almost every incident.

Why the parent breaker trips most

With use_real_memory at its default of true, the parent breaker compares the current real heap usage plus the new reservation against 95% of the heap. Real heap usage includes garbage that has not been collected yet. So on a node with a high heap floor, the parent breaker can trip on a perfectly ordinary request because the heap happened to be near the top of its sawtooth.

Two things follow from that. First, a parent trip on a node with a low heap floor is usually a single expensive request. Second, a parent trip on a node with a high floor is a capacity or retention problem, and the request named in the error is often just the unlucky one that arrived at the wrong time.

Reading a circuit_breaking_exception

A typical error body looks like this:

{
  "error": {
    "type": "circuit_breaking_exception",
    "reason": "[parent] Data too large, data for [<http_request>] would be [30983784448/28.8gb], which is larger than the limit of [30601641984/28.5gb], real usage: [30983781376/28.8gb], new bytes reserved: [3072/3kb], usages [request=0/0b, fielddata=11817160704/11gb, in_flight_requests=3072/3kb, model_inference=0/0b, eql_sequence=0/0b, accounting=412938211/393.8mb]",
    "bytes_wanted": 30983784448,
    "bytes_limit": 30601641984,
    "durability": "PERMANENT"
  },
  "status": 429
}

Read it field by field:

• [parent] names the breaker that tripped. If it says [fielddata] or [request], the child limit was hit directly.
• data for [<http_request>] is the label of the reservation. For the parent breaker this is often just the request being received, not the operation that is filling memory.
• real usage against limit tells you how close to the ceiling the heap was.
• new bytes reserved: [3kb] is tiny here. That is the signature of "the heap was already full; this request just tipped it over".
• usages [...] is the most useful part. In this example fielddata holds 11 GB. That is your lead: something is loading fielddata, very likely an aggregation or sort on a text field with fielddata: true, or large global ordinals.
• durability says what kind of memory the breaker is full of. The request and in-flight breakers are TRANSIENT: their memory is released as requests complete, so a retry later may succeed. The fielddata breaker is PERMANENT: that memory stays loaded until someone intervenes (changes the query or mapping, or clears the cache). A parent trip takes the durability of whichever side, transient or permanent child usage, is larger; here fielddata dominates, so it is PERMANENT.

The HTTP status is 429, so well-behaved clients treat it as backpressure and retry with backoff. That is reasonable for TRANSIENT trips; for PERMANENT ones, retrying alone will keep failing until the retained memory is dealt with.

Reading _nodes/stats/breaker

To see every breaker on every node:

GET _nodes/stats/breaker

The shape of each breaker in the response:

{
  "fielddata": {
    "limit_size_in_bytes": 11166914969,
    "limit_size": "10.4gb",
    "estimated_size_in_bytes": 10937411942,
    "estimated_size": "10.1gb",
    "overhead": 1.03,
    "tripped": 214
  }
}

• estimated_size against limit_size is current headroom. Fielddata at 10.1 of 10.4 GB means any new fielddata load will trip.
• tripped is a cumulative count since node start. Like GC counters, it is useful as a difference over time. A breaker that tripped 214 times last month and has not moved today is history; one that moved 50 times in the last ten minutes is your incident.

For the parent breaker, estimated_size reflects real heap usage when use_real_memory is enabled.

To find which fields hold fielddata, and on which nodes:

GET _cat/fielddata?v&s=size:desc

Or with per-field detail for each node:

GET _nodes/stats/indices/fielddata?fields=*&filter_path=nodes.*.name,nodes.*.indices.fielddata

A field name you did not expect at the top of this list is usually the whole answer.

What actually causes heap pressure

The list of root causes is short, and each has a distinctive signature.

Fielddata on text fields

text fields have no doc values. Sorting or aggregating on one requires fielddata: true, which builds an in-heap structure of every unique term in every segment. On a high-cardinality text field this is enormous and grows with the data. Signature: the fielddata breaker's estimated_size is high, and _cat/fielddata names a text field.

Fix: aggregate on a keyword field or keyword sub-field instead (see mapping design), set fielddata back to false, and clear the cache:

POST my-index/_cache/clear?fielddata=true

Clearing the cache is relief, not a fix. The next query that sorts on that field reloads it.

Global ordinals on high-cardinality keyword fields

terms aggregations on keyword fields use global ordinals, a per-shard mapping from each segment's ordinals to a shard-wide ordinal. They live on the heap, are counted against the fielddata breaker, and are rebuilt after refreshes when the field is aggregated. On a field with tens of millions of unique values, global ordinals are large. eager_global_ordinals: true moves the cost to refresh time; it does not remove it.

Large or deep aggregations

The request breaker accounts for aggregation buckets. A terms aggregation with a large size, nested under a date_histogram, nested under another terms, multiplies bucket counts. The cluster-wide guard is search.max_buckets (default 65,536 in 8.x), which fails a response with too many buckets. Signature: request breaker trips, or too_many_buckets_exception, tied to dashboards or reports. See aggregations at scale for composite and pre-aggregation alternatives.

Oversized requests and responses

Very large bulk requests, size values in the tens of thousands, and huge terms query arrays all allocate on the coordinating node. The in-flight requests breaker tracks request bytes in transit; the response side allocates heap when the coordinating node merges results from many shards. index.max_result_window (default 10,000) stops from + size from going past that point; use search_after with a point in time for deep pagination.

Too many shards and too many fields

Each shard carries fixed overhead: Lucene structures, per-field metadata, segment bookkeeping. Recent versions have greatly reduced per-segment heap use, but thousands of tiny shards on a node still add up, and so does a mapping with tens of thousands of fields repeated across many indices. These show up as a rising heap floor and a growing accounting breaker, not as single-request trips. The fix is fewer, larger shards: see shard sizing.

Indexing load

The indexing buffer (indices.memory.index_buffer_size, default 10% of heap) is shared among actively indexing shards. Separately, indexing pressure (indexing_pressure.memory.limit, default 10% of heap) caps the bytes of outstanding indexing requests on a node and rejects new ones beyond it. Rejections here appear in:

GET _nodes/stats/indexing_pressure

This is a heap guard specific to the write path. See indexing performance for how bulk sizing interacts with it.

Caches that are working as designed

The node query cache (indices.queries.cache.size, default 10% of heap) and the shard request cache (indices.requests.cache.size, default 1% of heap) are bounded. They contribute to the floor but do not grow without limit. Do not chase them first.

Failure modes and anti-patterns

The last row is the anti-pattern worth stating plainly: raising a breaker limit converts a failed request into a dead node. The defaults are set to leave room for the garbage collector to work. If you raise indices.breaker.total.limit to 99%, you will see fewer 429s right up until the JVM throws OutOfMemoryError, and by default the node writes a heap dump and the process exits. A failed request is an incident; a node exiting under load can start a cascade as its shards' load moves to the remaining nodes.

Two more anti-patterns:

• Switching garbage collectors or tuning G1 flags from a blog post. The bundled defaults are tested. Change them only with GC logs before and after.
• Disabling use_real_memory. It makes the parent breaker blind to real heap usage and lowers its limit to 70% of an estimate. You lose the most accurate guard you have.

Runbook: a node is past 85% heap

Work through this in order. Each step either finds the cause or narrows it.

1. Find which nodes and how bad.

   http GET _cat/nodes?v&h=name,node.role,heap.percent,heap.max,ram.percent,cpu,load_1m&s=heap.percent:desc

   One node high suggests a hot shard or an uneven workload. All data nodes high suggests cluster-wide load or retention.

2. Separate floor from peak. Pull _nodes/stats/jvm twice, 60 seconds apart. Compute old-gen used_in_bytes and the change in GC counts and times. A high floor plus old collections counting up is real pressure.

3. Read the breakers.

   http GET _nodes/stats/breaker?filter_path=nodes.*.name,nodes.*.breakers.*.estimated_size,nodes.*.breakers.*.limit_size,nodes.*.breakers.*.tripped

   Note which breakers are near their limit and which tripped counters are moving.

4. If fielddata is large, run GET _cat/fielddata?v&s=size:desc, identify the field, find the dashboard or query that sorts or aggregates on it, and change it. Clear fielddata only after the source is stopped.

5. If the request breaker is tripping, find the expensive searches that are running now:

   http GET _tasks?actions=*search*&detailed=true&group_by=parents

   Long-running tasks with large aggregations in their description are your suspects. Cancel the worst if it is safe to do so:

   http POST _tasks/<task_id>/_cancel

6. Check the shard count on the hot node.

   http GET _cat/allocation?v

   A node with far more shards than its peers carries more fixed overhead.

7. Check what the node is actually doing.

   http GET _nodes/<node_name>/hot_threads

   Threads stuck in GC, aggregation or merge code tell you which subsystem is busy.

8. Only after the cause is known, decide on capacity: add data nodes if the floor is high because of genuine load and data volume, or fix the query, mapping or shard count if it is not.

9. Confirm the fix. Repeat steps 2 and 3. The floor should fall, and the tripped counters should stop moving.

A short checklist for new clusters

• Heap equal for -Xms and -Xmx, set through jvm.options.d or left to automatic sizing.
• Heap no more than 50% of RAM or the container limit.
• Startup log shows compressed ordinary object pointers [true].
• Swap disabled, or bootstrap.memory_lock: true verified with mlockall.
• Circuit breaker settings at defaults.
• No text field mapped with fielddata: true unless you have measured the cost.
• Alerting on old-generation usage after GC, not on raw heap.percent.
• Alerting on the change in tripped per breaker, not the cumulative value.
• Shard counts per node within the guidance in shard sizing.

Key takeaways

• The heap is a ceiling: no more than half of RAM, under the compressed-oops threshold, and the rest goes to the filesystem cache that search depends on.
• Heap usage is a sawtooth. The signal is old-generation occupancy after collections and how fast GC time is growing.
• A circuit breaker trip is a node protecting itself. Read the usages list and durability to find the cause.
• The parent breaker with real-memory accounting trips on whatever request arrives when the heap is full, so the named request is not always the culprit.
• Fielddata on text, high-cardinality global ordinals, deep aggregations, oversized requests and too many shards account for most heap pressure.
• Raising breaker limits trades a 429 for an OutOfMemoryError. Leave them at their defaults.

How Sparced helps

Sparced reads node stats, thread pools and circuit breakers for every node in one pass, so you see heap usage and each breaker's estimated size, limit and trip count side by side without stitching together several API calls. When a node is past 85% heap it makes further calls to gather more evidence about that node, and the finding names the node, shows the numbers it read, and gives the commands to check and confirm. It does not change breaker limits or heap settings, and it does not guess which query caused a trip when the cluster's own output does not say. It installs no agent and writes nothing to the cluster.