A Logstash deployment fails in three predictable ways: it cannot keep up (throughput), it loses events when something restarts or a destination goes away (durability), or one slow destination stalls everything else (coupling). Each has a specific lever. Throughput is governed by pipeline.workers and pipeline.batch.size and by the cost of your filters. Durability comes from the persistent queue for transient failures and the dead letter queue for events Elasticsearch refuses outright. Coupling is solved structurally, by splitting one pipeline into several connected pipelines using the patterns Logstash documents: distributor, output isolator, forked path and collector. This article explains the execution model underneath, gives the sizing arithmetic, and shows how to read the monitoring API to know which of the three problems you have.
The mental model: inputs, a queue, and a pool of workers
Every Logstash pipeline has the same shape:
inputs -> queue -> [ worker threads: filters -> outputs ]
- Inputs run in their own threads and push events into the pipeline's queue. They do not run filters.
- The queue sits between inputs and workers. It is either in memory (the default) or on disk (persistent).
- Workers take a batch of events from the queue, run that batch through every filter in order, then hand the batch to the outputs. A worker does not take its next batch until the outputs have accepted the current one.
That last point is the whole story of backpressure in Logstash. When an output is slow (Elasticsearch returning 429, a Kafka broker unavailable), the worker holding that batch waits. When all workers wait, nothing is taken from the queue. When the queue is full, inputs block. What inputs do when they block depends on the protocol: Beats and Elastic Agent inputs signal back to the shipper, which stops sending and buffers or retries; a TCP input stops reading from its sockets; a UDP or syslog-over-UDP input cannot push back, so events are dropped at the socket.
Every event in flight sits on the JVM heap while it is in a batch. That gives the first sizing number:
in-flight events = pipeline.workers x pipeline.batch.size
Workers, batch size and batch delay
Three settings govern the worker loop. They can be set globally in logstash.yml or per pipeline in pipelines.yml.
| Setting | Default | What it controls |
|---|---|---|
pipeline.workers |
number of CPU cores on the host | Threads running filters and outputs in parallel |
pipeline.batch.size |
125 | Maximum events a worker takes per batch |
pipeline.batch.delay |
50 ms | How long a worker waits for a batch to fill before processing a partial one |
Workers
Filters are CPU work. Grok with complex patterns, Ruby filters, translate lookups and JSON parsing all consume cores. The default of one worker per core is right for a CPU-bound pipeline. Two cases justify more workers than cores:
- Outputs that block on network I/O for long periods. While a worker waits on an output, its core is idle. Adding workers can keep cores busy, but only if the destination can actually absorb more concurrent requests.
- Filters that call out (an
httpfilter, a JDBC or Elasticsearch lookup). Same reasoning: the thread waits, the core does not.
If CPU is already saturated, more workers add context switching and nothing else. If CPU is low and throughput is low, the bottleneck is downstream, not in workers.
Order matters for some pipelines. With more than one worker, events can be processed out of order. pipeline.ordered defaults to auto, which preserves order only when there is a single worker. If you rely on order (aggregate filter, some multi-line reconstructions), you are limited to one worker for that pipeline, which is a strong argument for isolating it in a pipeline of its own.
Batch size
The batch size is also, in practice, the bulk request size the Elasticsearch output sends: each worker's batch goes out as a bulk request (the output splits very large batches into more than one request). The default of 125 events is conservative. Larger batches amortize per-request overhead in both Logstash and Elasticsearch.
A common rule of thumb is to raise pipeline.batch.size towards a few hundred to a couple of thousand events, measuring as you go, until either throughput stops improving or the resulting bulk requests become large in bytes. Bulk shape matters on the Elasticsearch side too; see indexing performance.
Worked example: heap and bulk size
Assume:
- 16 cores, so 16 workers by default
pipeline.batch.size: 1000- average event size after filtering: 2 KB, but Logstash's in-memory representation of an event is larger than its serialized JSON (field objects, metadata)
In-flight events: 16 x 1,000 = 16,000.
Serialized size in flight: 16,000 x 2 KB = 32 MB. The heap cost is a multiple of that. A conservative rule of thumb is to budget several times the serialized size, so plan on the order of 100 to 200 MB of heap just for batches, before plugins, caches and the persistent queue's buffers.
Bulk request size: each batch is about 1,000 x 2 KB = 2 MB per bulk. Reasonable. With 16 workers you have up to 16 concurrent bulk requests to the cluster from one Logstash instance. Ten such instances mean 160 concurrent bulks, which is the number Elasticsearch's write pool actually sees. See thread pools and rejections.
Logstash's own guidance is that heap for typical ingest workloads should usually be no less than 4 GB and no more than 8 GB; the shipped default in jvm.options is lower. Set -Xms and -Xmx equal. More heap than 8 GB rarely helps and lengthens GC pauses; if you need more throughput, add instances.
Batch delay
pipeline.batch.delay only matters at low volume. If events trickle in, a worker waits up to 50 ms for its batch to fill before processing a partial batch. At high volume batches fill instantly and the delay never applies. Leave it alone unless you are tuning latency for a very low-volume pipeline.
Persistent queues
The default memory queue is fast and bounded (it holds only what workers are processing), but everything in it is lost if the Logstash process dies. A persistent queue (PQ) writes events to disk when the input receives them and removes them only when the outputs have acknowledged them.
Enable it per pipeline:
- pipeline.id: beats-ingest
path.config: "/etc/logstash/conf.d/beats.conf"
queue.type: persisted
queue.max_bytes: 20gb
The settings that matter:
| Setting | Default | Meaning |
|---|---|---|
queue.type |
memory |
memory or persisted |
queue.max_bytes |
1024mb | Total disk the queue may use; when reached, inputs are blocked |
queue.max_events |
0 (unlimited) | Alternative cap by count |
queue.page_capacity |
64mb | Size of each page file on disk |
queue.checkpoint.writes |
1024 | Events written between forced checkpoints |
queue.drain |
false | Whether to empty the queue before shutting down |
path.queue |
under path.data |
Where queue files live |
What a PQ does and does not protect against
It protects against:
- Logstash process restarts and crashes. Events accepted by the input but not yet output are replayed on start. Delivery is at least once: an event that was output just before a crash may be output again after restart. Use deterministic document IDs (a fingerprint of the event) if duplicates matter.
- Downstream outages, up to
queue.max_bytes. While Elasticsearch is down, the queue absorbs incoming events and inputs keep accepting until it is full. - Bursts larger than processing capacity.
It does not protect against:
- Disk failure on the Logstash host. A PQ is local storage. If you need durability across host loss, the buffer belongs in Kafka or the shipper. See a logging architecture with a Kafka buffer.
- Inputs that cannot be replayed (UDP, push-based HTTP without client retry) before the event reaches the queue.
- Loss between checkpoints on power failure. With
queue.checkpoint.writes: 1024, up to that many recently written events may not be durable if the host loses power. Setting it to 1 forces a checkpoint per event, which costs significant throughput. Most deployments accept the default.
Sizing the PQ
Size it to the outage you want to survive.
Assume an instance receives 8,000 events per second at 1.5 KB each, and you want to ride out a 2-hour Elasticsearch maintenance window without blocking shippers:
8,000 events/s x 1.5 KB = 12,000 KB/s, about 11.7 MB/s
11.7 MB/s x 7,200 s = about 84,000 MB, about 82 GB
The on-disk representation is not identical to the raw event size, so add a margin; a rule of thumb is 20 to 30%. Set queue.max_bytes to about 100 GB, and make sure path.queue is on a disk with that much free space plus room for the OS and logs. Also budget the time to drain: when Elasticsearch returns, Logstash must process 82 GB of backlog plus live traffic. If its peak throughput is 12,000 events per second against 8,000 live, it drains the backlog at 4,000 events per second net, which is 57.6 million events in about 4 hours. The cluster must also be able to absorb that elevated rate.
The dead letter queue
A PQ handles transient failures. Some failures are not transient: Elasticsearch rejects a document because of a mapping conflict (a field that is a long in the index arrives as an object) or another error that will never succeed on retry. By default, the Elasticsearch output logs these and drops the event. The dead letter queue (DLQ) keeps them instead.
dead_letter_queue.enable: true
dead_letter_queue.max_bytes: 2gb
path.dead_letter_queue: /var/lib/logstash/dlq
dead_letter_queue.enable is false by default; dead_letter_queue.max_bytes defaults to 1024mb. When the DLQ is full, the default policy drops new entries; recent releases add an option to drop the oldest instead, and age-based retention. Check the settings reference for your version before relying on those.
Important details:
- The DLQ receives events the Elasticsearch output gets a 400 or 404 for, such as mapping errors. It does not receive 429s or other retryable responses; those are retried, which is backpressure working as intended.
- Each DLQ entry stores the original event plus metadata: the plugin, the reason, and the timestamp.
- The DLQ is per pipeline, under
path.dead_letter_queue/<pipeline_id>.
To reprocess, run a separate pipeline that reads the DLQ, fixes the event, and sends it on:
input {
dead_letter_queue {
path => "/var/lib/logstash/dlq"
pipeline_id => "beats-ingest"
commit_offsets => true
}
}
filter {
# the reason is in [@metadata][dead_letter_queue][reason]
mutate { rename => { "[http][response][body]" => "[http][response][body_text]" } }
}
output {
elasticsearch {
hosts => ["https://es01:9200"]
data_stream => "true"
}
}
commit_offsets => true records progress so a restart does not reprocess the same entries. An unmonitored DLQ is a bucket of silently lost data: alert on its size growing, and review the reasons regularly. Mapping conflicts are usually a template problem worth fixing at the source; see mapping design.
Pipeline-to-pipeline patterns
One Logstash instance can run many pipelines, defined in pipelines.yml, each with its own workers, batch size and queue. Pipelines can send events to each other in memory with the pipeline input and output:
# sending pipeline
output { pipeline { send_to => ["es-out"] } }
# receiving pipeline
input { pipeline { address => "es-out" } }
Each pipeline is independent: separate queue, separate worker pool, separate failure behavior. That independence is what makes the four documented patterns work.
Distributor
One input receives mixed traffic and routes it to type-specific pipelines.
- pipeline.id: beats-in
config.string: |
input { beats { port => 5044 } }
output {
if [data_stream][dataset] == "nginx.access" { pipeline { send_to => ["nginx"] } }
else if [data_stream][dataset] == "system.syslog" { pipeline { send_to => ["syslog"] } }
else { pipeline { send_to => ["fallback"] } }
}
- pipeline.id: nginx
path.config: "/etc/logstash/conf.d/nginx.conf"
- pipeline.id: syslog
path.config: "/etc/logstash/conf.d/syslog.conf"
- pipeline.id: fallback
path.config: "/etc/logstash/conf.d/fallback.conf"
Why it helps: one huge config with deeply nested conditionals is hard to reason about and runs every event through every conditional. Separate pipelines can each have their own worker count, so the expensive nginx grok pipeline gets more workers than the cheap fallback.
Output isolator
A pipeline with two outputs is only as fast as the slower one: workers wait until every output accepts the batch. If Elasticsearch is fine but the archive output to object storage stalls, Elasticsearch ingest stops too.
The output isolator splits each destination into its own pipeline, each with its own persistent queue:
- pipeline.id: intake
config.string: |
input { beats { port => 5044 } }
output { pipeline { send_to => ["es-out", "archive-out"] } }
- pipeline.id: es-out
queue.type: persisted
queue.max_bytes: 50gb
config.string: |
input { pipeline { address => "es-out" } }
output { elasticsearch { hosts => ["https://es01:9200"] data_stream => "true" } }
- pipeline.id: archive-out
queue.type: persisted
queue.max_bytes: 50gb
config.string: |
input { pipeline { address => "archive-out" } }
output { s3 { bucket => "log-archive" } }
Now if the archive output stalls, its PQ fills while Elasticsearch ingest carries on. Only when the archive PQ is full does backpressure reach the intake pipeline, and the time until that happens is queue.max_bytes divided by the event rate. Size each PQ for the longest outage you expect on that destination.
Forked path
The same event goes to several pipelines that each process it differently. For example, the full event is indexed into Elasticsearch, while a stripped copy with personal fields removed goes to a third-party destination:
output { pipeline { send_to => ["full-es", "redacted-partner"] } }
Each receiving pipeline gets its own copy, so mutating fields in one does not affect the other. This is cleaner than clone filters plus conditionals in one pipeline.
Collector
The reverse of the distributor: several pipelines send to one shared output pipeline, so the Elasticsearch connection settings, credentials and output tuning live in one place. Useful when many teams own input-and-filter pipelines but the platform team owns the output.
When to split a pipeline
| Situation | Pattern | Benefit | Cost |
|---|---|---|---|
| Mixed sources, different parsing costs | Distributor | Per-type workers, simpler configs | More pipelines to monitor |
| Two or more destinations with different reliability | Output isolator | One outage does not stall the other | Disk for one PQ per destination |
| Same events, different transformations per destination | Forked path | Independent mutation | Events copied, more CPU and memory |
| Many producers, one managed destination | Collector | Single place for output config | Shared bottleneck at the output |
| Ordered processing required for one source | Dedicated pipeline with 1 worker | Order preserved without slowing the rest | Low throughput for that pipeline |
Two cautions. First, the pipeline output to an address whose receiving pipeline is not running blocks, the same as any blocked output; ensure all pipelines start together. Second, if the sending pipeline uses a memory queue, events between it and a persistent downstream queue are not durable. Put the PQ where durability matters, usually at the intake (to absorb shipper bursts) and at each isolated output.
Monitoring pipeline stats
Logstash exposes a monitoring API, by default on localhost at the first free port from 9600:
curl -s localhost:9600/_node/stats/pipelines/beats-ingest?pretty
The fields that answer the three questions:
{
"pipelines": {
"beats-ingest": {
"events": {
"in": 51230011,
"filtered": 51229100,
"out": 51228900,
"duration_in_millis": 9811233,
"queue_push_duration_in_millis": 120044
},
"plugins": {
"filters": [
{ "id": "grok-nginx", "name": "grok", "events": { "in": 51229900, "out": 51229100, "duration_in_millis": 7120334 } }
],
"outputs": [
{ "id": "es-main", "name": "elasticsearch", "events": { "in": 51229100, "out": 51228900, "duration_in_millis": 1840221 } }
]
},
"queue": { "type": "persisted", "events_count": 412900 }
}
}
}
How to use them:
- Throughput: sample
events.outtwice and divide by the interval. - Which plugin is slow: divide each plugin's
duration_in_millisby itsevents.outto get milliseconds per event. Here grok spends 7,120,334 ms on 51.2 million events, about 0.14 ms per event, and dominates the pipeline. Give plugins explicitids in your config, or the stats show generated ones you cannot map back to lines. - Backpressure from outputs: a rising
queue.events_counton a persistent queue, and outputduration_in_millisgrowing faster than events, mean outputs are the bottleneck. Check Elasticsearch for rejections. - Inputs blocked: a growing
queue_push_duration_in_millismeans inputs spend time waiting to push into a full queue.
Recent 8.x releases also report flow metrics per pipeline, such as input, filter and output throughput, queue backpressure and worker utilization, as rates over several windows. Pipeline worker_utilization near 100 means every worker is busy, and time a worker spends waiting on an output counts as busy, so saturation can be either expensive filters or a slow destination; the per-plugin worker_utilization and milliseconds per event tell you which. A value near 0 means the pipeline is starved: inputs are not supplying enough data.
For CPU, the hot threads API names the threads that are busy:
curl -s "localhost:9600/_node/hot_threads?human=true&threads=5"
For long-term visibility, collect these metrics into a monitoring cluster with Metricbeat or Elastic Agent's Logstash integration and view them in Stack Monitoring. See monitoring Elasticsearch in production.
Failure modes
Throughput stuck well below expectation, CPU low. Cause: outputs blocking. Confirm: worker utilization high while CPU is low, the output plugin's duration dominant, and Elasticsearch shows write rejections or high indexing latency. Fix: on the Elasticsearch side, not in Logstash; adding workers just adds concurrent bulks to an overloaded cluster.
Throughput stuck, CPU pinned. Cause: an expensive filter. Confirm with per-plugin milliseconds per event and hot threads. Fix: anchor grok patterns, replace grok with dissect where the format is fixed, move lookups to a cached translate, or add instances. The trade-offs of where to parse are in ingest pipelines vs Logstash.
Events lost on restart. Cause: memory queue. Confirm: queue.type is memory in the pipeline stats. Fix: persistent queue for any pipeline whose inputs cannot replay.
One destination outage stops all ingest. Cause: multiple outputs in one pipeline. Fix: output isolator with a PQ per destination.
Events silently missing from Elasticsearch with no errors visible. Cause: mapping errors with the DLQ disabled, logged and dropped. Confirm: Logstash log lines about failed indexing with status 400. Fix: enable the DLQ, fix the template, reprocess.
Disk full on the Logstash host. Cause: PQ or DLQ sized larger than the disk, or the DLQ never drained. Fix: size queue.max_bytes and dead_letter_queue.max_bytes against the actual free space, and alert on both.
Long GC pauses, erratic throughput. Cause: workers times batch size too large for the heap, or heap set above 8 GB. Confirm with JVM stats in _node/stats/jvm. Fix: lower batch size or workers, keep heap within 4 to 8 GB, scale out.
Runbook: Logstash is falling behind
- Is it Logstash or downstream? Check
GET _node/stats/pipelinesfor worker utilization (flow metrics) and per-plugin time. If outputs dominate, go to the cluster. - Check the cluster. Write rejections, indexing pressure rejections, node heap and disk. Do not add workers while the cluster is rejecting.
- If filters dominate, find the plugin with the highest milliseconds per event and fix it, or add instances.
- Check queue state.
queue.events_countrising means input rate exceeds output rate. Estimate time to full: free queue bytes divided by the net inflow rate. - Protect the shippers. If the PQ will fill before the problem is fixed, decide what gives: shippers buffering (Beats and Agent back off), or a temporary drop of low-value data at the distributor.
- Check the DLQ. Growth means mapping errors, which no amount of capacity fixes.
- After recovery, compute drain time and confirm the cluster can take the extra rate.
Key takeaways
- A worker holds its batch until every output accepts it; that is where backpressure comes from and why one slow output stalls a pipeline.
- In-flight events are
pipeline.workers x pipeline.batch.size. Defaults are cores and 125; heap for typical workloads is 4 to 8 GB. - Persistent queues give at-least-once delivery across restarts and absorb outages up to
queue.max_bytes, not host loss. - The DLQ catches 400 and 404 rejections like mapping conflicts, not 429s. It is off by default and must be monitored.
- Use the distributor, output isolator, forked path and collector patterns to give each concern its own workers and queue.
- Read per-plugin milliseconds per event and worker utilization before changing any setting.
How Sparced helps
Sparced inspects Elasticsearch, not Logstash, so it will not read your pipeline stats. What it does show is the downstream half of most Logstash slowdowns: in one read-only pass it reports write thread pool rejections per node, nodes past 85% heap, and nodes near their disk watermarks, each with the evidence and a fix command with the node names filled in. That tells you quickly whether a Logstash backlog is a cluster problem, so you do not add workers to a pipeline whose real bottleneck is a saturated hot node.