Elasticsearch Thread Pools and Rejections: What 429 Really Means

A thread pool rejection is Elasticsearch telling you that a node received work faster than it could do it, waited as long as it was configured to wait, and then refused the rest. The 429 Too Many Requests and the es_rejected_execution_exception behind it are not the problem; they are the cluster protecting its heap and its latency from a problem upstream. The fix is almost never a bigger queue. It is one of three things: send less concurrent work (client backpressure and retry), make each unit of work cheaper (fewer shards per request, cheaper queries, better bulk shape), or add capacity where the saturation actually is (a hot node, not the cluster average). This article explains how the pools work, how to read them, how to do the capacity arithmetic, and how to tell which of the three fixes you need.

The mental model: fixed pools, bounded queues, shard-level tasks

Each Elasticsearch node runs a set of named thread pools. Each pool has a type:

• fixed: a fixed number of threads and a bounded queue. When all threads are busy, new tasks wait in the queue. When the queue is full, new tasks are rejected. write, search and get are fixed pools.
• scaling: threads are created on demand up to a maximum and retired after a keep-alive. Scaling pools do not reject in the same way; work waits for a thread. management, generic, refresh, flush and snapshot are scaling pools.

The key point most people miss is what a task is. A task in the search pool is not a search request. It is one shard-level operation. A search against an index with 30 primary shards becomes 30 shard-level tasks, spread across the nodes that hold a copy of each shard, each one landing in that node's search queue. A _bulk request that touches documents in 20 shards becomes up to 20 shard-level bulk tasks on the primaries' write pools, and then the same again on the replicas' write pools.

So the load on a pool is:

tasks per second on a node = requests per second x shards touched per request / nodes sharing those shards

This is why shard count, not request count, is usually the variable that explains rejections. Doubling shards for an index doubles the search tasks each query generates. It is also why a single node can reject while the cluster average looks healthy: if a hot index's shards concentrate on two nodes, those two nodes take the whole fan-out.

The pools that matter and their defaults

Defaults are derived from the node's allocated processors, which Elasticsearch detects from the OS (respecting container CPU limits) and which you can override with node.processors. On a node with 8 allocated processors:

The queue defaults moved in 9.x. On 8.x the search queue is a fixed 1,000; since 9.0 it is 1,000 times the pool size. The write queue is 10,000 through 9.1; in later 9.x it is the larger of 10,000 and 750 times the allocated processors, and since 9.1 a separate write_coordination pool handles the coordinating side of bulk requests. Check queue_size in _cat/thread_pool rather than assuming.

Two things to notice. The search pool is oversized relative to cores on purpose, because search tasks spend part of their time waiting on I/O. The write pool is sized to cores because indexing is CPU-bound: tokenization, Lucene indexing, and merges competing for the same cores.

Thread pool settings are static node settings in elasticsearch.yml (for example thread_pool.write.queue_size). Changing them requires a node restart. That alone should make you hesitate before treating them as a tuning knob.

If you run Elasticsearch in containers and the container is limited to 4 CPUs on a 64-core host, check that the node actually allocated 4 processors. If it allocated 64, the write pool has 64 threads fighting for 4 cores and latency is terrible long before any queue fills:

GET _nodes/os?filter_path=nodes.*.name,nodes.*.os.allocated_processors,nodes.*.os.available_processors

Reading the pools

The fast view is _cat/thread_pool. Ask for the columns you need and filter to the pools that reject:

GET _cat/thread_pool/write,search,get?v=true&h=node_name,name,size,active,queue,queue_size,rejected,completed&s=rejected:desc

node_name  name    size active queue queue_size rejected completed
hot-03     write      8      8  9874      10000    41233  981233411
hot-04     write      8      8  7712      10000    18409  979120334
hot-01     write      8      3     0      10000        0  980122006
hot-03     search    13     13   940       1000     2210   44211233

How to read each column:

• size: configured thread count. Confirms the pool is sized as you think.
• active: threads busy right now. active == size means the pool is saturated at this instant.
• queue: tasks waiting. A persistently non-zero queue means sustained saturation, not a burst.
• queue_size: the bound. When queue reaches it, the next task is rejected.
• rejected: cumulative since the node started. A large number on its own tells you nothing about now. You need the change over an interval.
• completed: also cumulative, useful as a throughput denominator.

In the example, hot-03 and hot-04 are saturated and rejecting, while hot-01 in the same tier is idle. That is not a capacity problem in the cluster. It is a placement problem: something hot lives on two nodes.

To get rates, sample twice and subtract, or read the same counters from node stats:

GET _nodes/stats/thread_pool?filter_path=nodes.*.name,nodes.*.thread_pool.write,nodes.*.thread_pool.search

A monitoring system that graphs rejected as a rate per node per pool is the single most useful view here. An alert on "rejections increased by more than zero in the last five minutes" is a reasonable page for the write pool on a logging cluster where data loss is the risk, and a reasonable ticket for search, where the client sees an error and can retry. See monitoring Elasticsearch in production for where to set thresholds.

What a rejection looks like to the client

Write rejections in a bulk request

A _bulk request is not rejected as a whole. The coordinating node splits it per shard, and each shard-level request can succeed or be rejected independently. The bulk response returns HTTP 200 with "errors": true, and individual items carry status 429:

{
  "took": 412,
  "errors": true,
  "items": [
    { "index": { "_index": "logs-app-default", "_id": "a1", "status": 201, "result": "created" } },
    { "index": {
        "_index": "logs-app-default",
        "_id": "a2",
        "status": 429,
        "error": {
          "type": "es_rejected_execution_exception",
          "reason": "rejected execution of ... on EsThreadPoolExecutor[name = hot-03/write, queue capacity = 10000, ...]"
        }
    } }
  ]
}

This is the most common source of silent data loss in custom indexers: the code checks the HTTP status, sees 200, and moves on. Always check errors and walk the items. The reason names the node and the pool, which tells you exactly where the saturation is.

Indexing pressure, the other write rejection

Since 7.9, Elasticsearch also limits the bytes of indexing work outstanding on a node, independent of the queue: indexing_pressure.memory.limit, which defaults to 10% of the heap. When in-flight indexing bytes for coordinating, primary and replica stages exceed it, new work is rejected with a 429 even if the write queue is not full. This is a heap protection, and it often fires first when individual bulk requests are very large.

GET _nodes/stats/indexing_pressure?filter_path=nodes.*.name,nodes.*.indexing_pressure.memory.total

Look at coordinating_rejections, primary_rejections and replica_rejections. If these increase while _cat/thread_pool shows write queues well below 10,000, your bulks are too large in bytes, or a few nodes coordinate too much of the traffic, not too many in count.

Search rejections

For search, a rejection happens per shard. With the default allow_partial_search_results: true, the search returns 200 with partial results and a non-zero _shards.failed:

{
  "_shards": {
    "total": 30,
    "successful": 27,
    "skipped": 0,
    "failed": 3,
    "failures": [
      { "shard": 4, "index": "orders", "node": "...", "reason": { "type": "es_rejected_execution_exception" } }
    ]
  }
}

Dashboards that ignore _shards.failed show numbers that are quietly wrong. If every shard is rejected, the request fails with a 429 or 503. For applications where a partial answer is worse than no answer, set allow_partial_search_results=false on the request so the failure is explicit.

The capacity arithmetic

Rejections are a queueing problem, and the arithmetic is simple enough to do on a whiteboard.

Search, worked example

Assume:

• 6 data nodes, 8 allocated processors each, so 13 search threads per node.
• One index, 12 primary shards with 1 replica, 24 shard copies, 4 per node.
• A typical query takes 20 ms per shard on average (measure with the profile API or slow logs).
• Each search hits all 12 shards once (one copy of each).

Capacity per node: 13 threads / 0.020 s = 650 shard tasks per second.

Capacity of the cluster: 6 x 650 = 3,900 shard tasks per second.

Each search costs 12 shard tasks, so the cluster can serve 3,900 / 12 = 325 searches per second at full saturation. In practice you want headroom; a common rule of thumb is to plan for no more than about 70% sustained utilization, which is about 225 searches per second.

Now see what the queue buys. At saturation a node drains 650 tasks per second. A full 8.x queue of 1,000 tasks takes 1,000 / 650 = 1.5 seconds to drain, so a task that enters the back of a full queue waits roughly 1.5 seconds before it starts. If you raised queue_size to 10,000, you would trade rejections for waits of about 15 seconds, which most clients time out on long before: the query still fails, but after holding heap for 15 seconds. On 9.x the default queue for 13 threads is already 13,000 (13,000 / 650 = 20 seconds), so expect saturation there to show up as latency and client timeouts before it shows up as rejections.

Now see what shard count costs. If the same data were split into 48 primaries instead of 12, each search would cost 48 shard tasks and the cluster would serve about 81 searches per second, a quarter of the capacity, for no gain in speed unless individual shards were too large. Oversharding is the most common root cause of search rejections in clusters that are not obviously busy. See shard sizing.

Write, worked example

Assume:

• 4 hot nodes, 16 allocated processors each, so 16 write threads per node.
• Clients send bulks of 1,000 documents.
• The target data stream's write index has 4 primaries and 1 replica.
• Each shard-level bulk of about 250 documents takes about 50 ms on a primary.

Each client bulk becomes 4 primary tasks (one per shard) plus 4 replica tasks, 8 write tasks in total across the 4 nodes, 2 per node.

Per node capacity: 16 threads / 0.050 s = 320 shard-level bulk tasks per second.

Per node load at 160 client bulks per second across the cluster: 160 x 2 = 320 tasks per second per node. You are at 100% saturation, and any burst fills the queue.

What helps:

• If the same 1,000 documents were routed to an index with 16 primaries (one per thread, which feels intuitive), each bulk would generate 32 small tasks per request across the cluster instead of 8. Per-task overhead grows and the queue fills faster. More shards per bulk is not free.
• Larger bulks that each touch the same number of shards amortize per-task overhead, until indexing pressure (10% of heap in flight) becomes the limit.
• Removing the replica during a one-off backfill halves write tasks. See indexing performance.

The general lesson: compute tasks per node per second from your shard layout before you touch any setting. It usually points straight at the cause.

Why raising the queue size is usually wrong

It is the first suggestion in every forum thread, and it is usually the wrong one. Here is why, in order of importance.

1. A queue does not add throughput. Throughput is threads divided by service time. A longer queue lets more work wait; it does not make any of it finish faster. Under sustained overload, any finite queue fills; a bigger one just takes longer to fill.
2. Waiting work holds memory. Every queued bulk task holds its documents on the heap. Ten thousand queued shard-level bulks of a few hundred KB each is gigabytes of heap. Raising the write queue moves the failure from a clean 429 to heap pressure, long GC pauses and eventually circuit breaker trips, which are far worse. See JVM heap and circuit breakers.
3. Waiting turns into latency, then timeouts. As computed above, a deep search queue converts rejections into multi-second waits. The client times out and often retries, adding more work to the same queue. You have built a retry storm.
4. It hides the signal. Rejections are the earliest and clearest overload signal you get. Suppressing them delays the moment you notice.

There are narrow cases where a modest queue change is reasonable: a small cluster with short, spiky bursts and plenty of heap, where the burst is known to drain in a second or two. Even then, treat it as a temporary measure and record why.

Backpressure on clients

A well-built client treats a 429 as an instruction to slow down, not as an error.

Retry only what failed. For bulk, collect the items with status 429 and resend just those. Resending the whole bulk repeats the operations that already succeeded (and, with auto-generated IDs, creates duplicate documents) and doubles the load at the worst moment.

Back off exponentially with jitter. A typical policy: wait 100 ms, then 200, 400, 800, up to a cap of a few seconds, with random jitter so a fleet of clients does not retry in lockstep.

Bound concurrency. The number of in-flight bulk requests per client is the real throttle. If you have 50 indexer processes each running 8 concurrent bulks, the cluster sees 400 concurrent bulks. Reducing that is often the entire fix.

Use the shipped behavior where it exists. Elastic's own shippers already back off on 429: Beats and Elastic Agent retry with backoff, and the Logstash Elasticsearch output retries 429 responses rather than dropping them. That retry stalls the Logstash pipeline, which then pushes back on inputs, which is exactly what backpressure is for. Make sure what sits in front of them can absorb the pause: a persistent queue in Logstash or a Kafka topic. See Logstash pipeline architecture and a logging architecture with a Kafka buffer.

Never treat a 429 as fatal in batch jobs. A reindex or backfill script that exits on the first rejection is a script that will fail at 3 a.m. Build retries in from the start.

For search, client backpressure looks different: limit concurrent searches from dashboards and batch reports, and cap fan-out on the request with max_concurrent_shard_requests (which defaults to 5 per node per request) when a single heavy search should not monopolize a node's search pool.

Finding what is actually saturating the pool

When a pool is saturated, find out what is running in it.

Hot threads shows where CPU is going on each node:

GET _nodes/hot-03/hot_threads?threads=5&interval=500ms

If the top threads are in write and the stack shows Lucene indexing or analysis, indexing is CPU-bound: heavy analyzers, many fields, or an ingest pipeline with expensive grok patterns on the same nodes. If they are in search and the stack shows scoring or aggregation code, a specific query pattern is to blame.

The task management API shows what is running and for how long:

GET _tasks?actions=*search*&detailed=true&group_by=parents

Long-running parent tasks with many children are your expensive fan-out queries. The description field often includes the indices and the query. If clients set X-Opaque-Id, it shows here too and identifies the application.

Shard placement explains why only some nodes reject:

GET _cat/shards/logs-app-*?v=true&h=index,shard,prirep,node,store&s=node

If the current write index of the busiest data stream has all its primaries on two nodes, those two nodes take all the primary write work. A common fix is setting index.routing.allocation.total_shards_per_node in the template so that new write indices spread across the tier.

Slow logs identify which queries are expensive per shard; see search latency.

Failure modes

Rejections on two nodes out of twenty. Symptom: _cat/thread_pool shows high queue and rising rejected on a few nodes only. Cause: hot shards concentrated there, often the current write index of a high-volume data stream, or a coordinating-heavy client pinned to those nodes. Confirm with _cat/shards for the write index and the client connection list. Fix: spread the write index with total_shards_per_node or more primaries, and send client traffic to all data or coordinating nodes rather than one.

Write rejections while queues look empty. Symptom: 429s from bulk, _cat/thread_pool write queue near zero. Cause: indexing pressure limits, not the queue. Confirm in _nodes/stats/indexing_pressure. Fix: smaller bulks in bytes, more coordinating capacity, or fewer concurrent bulks per client.

Search rejections on a lightly loaded cluster. Symptom: rejections at modest QPS. Cause: oversharded indices or wildcard patterns that touch hundreds of shards per search. Confirm by checking _shards.total in a typical response. Fix: fewer, larger shards; narrower index patterns; time-bounded queries that let pre-filtering skip shards.

A dashboard that randomly shows lower numbers. Symptom: totals vary between refreshes. Cause: partial results from search rejections. Confirm by inspecting _shards.failed in the request Kibana makes. Fix: reduce fan-out or concurrency, and treat any non-zero failed as a defect.

Rejections followed by a heap incident after someone raised the queue. Symptom: queue sizes recently increased in elasticsearch.yml; heap climbs, GC pauses lengthen, breakers trip. Cause: the queue now holds gigabytes of pending work. Fix: revert the queue size, then address the real cause.

Management pool starvation. Symptom: stats APIs and _cat calls become slow or time out, monitoring gaps appear. Cause: many concurrent monitoring or stats calls (several monitoring tools polling every few seconds on a large cluster). Fix: reduce the number and frequency of pollers, and avoid repeated expensive calls such as full _nodes/stats on very large clusters.

Runbook: rejections in progress

1. Which pool, which nodes. GET _cat/thread_pool/write,search,get?v=true&h=node_name,name,active,queue,rejected&s=rejected:desc. Take it twice, 60 seconds apart, and compare rejected.
2. Write rejections without full queues? Check GET _nodes/stats/indexing_pressure. If indexing pressure is rejecting, the cause is in-flight bytes.
3. A few nodes or all? If a few, list shards on them with _cat/shards ... &s=node and find the hot index. If all, the cluster is genuinely at capacity for this workload.
4. What is running. GET _nodes/<node>/hot_threads and GET _tasks?actions=*search*&detailed=true. Name the query or the indexer.
5. Confirm clients retry correctly. Check the indexer logs for 429 handling and whether whole bulks are being resent.
6. Relieve immediately. Reduce concurrency at the source (pause a backfill, cap a batch job, lower the Logstash workers sending to the cluster). Do not change queue sizes.
7. Fix the layout. Spread hot write indices, reduce shard counts for fan-out-heavy indices through the template on next rollover.
8. Then capacity. If every node is saturated after the above, add nodes to the tier that is saturated, and redo the tasks-per-second calculation to size it.
9. Confirm. Rejection rate is flat at zero over 15 minutes and queue returns to near zero between bursts.

Key takeaways

• A pool task is a shard-level operation. Rejections scale with shards touched per request, not with requests.
• rejected in _cat/thread_pool is cumulative. Alert on its rate, per node, per pool.
• Write rejections come from either a full queue (10,000 by default on 8.x) or indexing pressure (10% of heap in flight); check both.
• Bulk responses return 200 with item-level 429s. Check errors and retry only failed items.
• Raising queue_size adds latency and heap use, not throughput. It is almost never the fix.
• Compute tasks per node per second from your shard layout; it usually names the cause.
• Backpressure belongs in the client: bounded concurrency, exponential backoff with jitter, and a buffer in front of the shipper.

How Sparced helps

Sparced reads the thread pool statistics for every node in its single read-only pass and reports pools that have rejected work, naming the node and the pool rather than a cluster average, so a rejection problem concentrated on two hot nodes is visible as exactly that. In the same pass it reads the shard layout and node heap, so you can see a saturated node alongside the shards it holds and whether it is past 85% heap. Each finding comes with the evidence, a suggested next command with the node names filled in, and a check to confirm the change worked. Sparced does not change queue sizes or tune anything for you; it shows you where the pressure is so you can fix the cause.