Logging Architecture with Kafka and Elasticsearch: A Sizing Guide

A logging pipeline that writes straight from shippers into Elasticsearch has no slack: when the cluster slows down, is upgraded or goes red, every shipper backs up at once, and anything that cannot buffer locally drops data. Putting Kafka between the shippers and the cluster gives you that slack, plus replay and fan-out. It does not give you exactly-once delivery into Elasticsearch, it does not make a slow cluster fast, and it adds a distributed system you now have to run. This article explains when the buffer is worth it, how partitions map to consumers, how backpressure actually travels through the system, how to make duplicates harmless, and how to size brokers, partitions and consumers with the arithmetic shown.

Why a buffer, and when not to add one

The core argument for Kafka in a logging path is decoupling in time. Producers write to Kafka at whatever rate the sources generate; consumers read from Kafka at whatever rate Elasticsearch can absorb. The difference accumulates as consumer lag, bounded by Kafka's retention.

What that buys you:

What it costs:

• A Kafka cluster with its own capacity planning, upgrades, security and monitoring.
• Extra latency between event and searchability, usually small, but unbounded when lag builds.
• A second place where data can be lost if it is misconfigured.
• More moving parts in every incident.

When a buffer is not worth it: a single cluster with modest volume, shippers that already buffer to local disk (Filebeat and Elastic Agent both keep a registry of file offsets and simply re-read logs from disk after an outage), no requirement for replay, and no second consumer. Log files on disk are already a buffer as long as they are retained longer than your worst outage. The case for Kafka becomes strong when sources cannot buffer (syslog over UDP, network devices, serverless functions, application appenders that log over the network), when volume is high enough that a cluster outage would overflow local buffers, or when several systems need the same stream.

The reference architecture

sources -> shippers -> Kafka topics -> consumers -> Elasticsearch
                         |
                         +-> other consumers (SIEM, archive, stream jobs)

Each hop has one job:

1. Shippers (Filebeat, Elastic Agent with a Kafka output, Fluent Bit, application appenders, syslog relays) read sources and produce to Kafka. They do little or no parsing.
2. Kafka stores events durably, partitioned and replicated, for a retention period.
3. Consumers (Logstash, or Elastic Agent consuming from Kafka) read topics in consumer groups and write to Elasticsearch with the Bulk API.
4. Elasticsearch parses, if parsing is done in ingest pipelines, and indexes.

Where parsing lives is a separate decision covered in /guides/ingest-pipelines-vs-logstash/. A buffer does not force parsing into Logstash. A common, clean split is: Logstash consumes and routes, ingest pipelines parse.

Topic design

Topic layout decides how independently you can operate each data source.

The common choice is one topic per dataset family (firewall, web access, application logs, audit), with a separate topic for anything with a stricter retention or compliance requirement. Audit logs in particular should not share a topic, and therefore a lag, with debug logs.

Message keys decide partition placement. For logs, ordering across a whole topic is rarely needed; Elasticsearch orders by @timestamp at query time anyway. A null key lets the producer spread messages across partitions evenly. Keying by host or service preserves per-key order but creates hot partitions when one key is much louder than the others, and a hot partition can only be drained by one consumer.

Partitions and consumers: the one rule

Kafka's consumer model has one rule that governs everything downstream: within a consumer group, each partition is assigned to exactly one consumer at a time. Consumers beyond the partition count sit idle.

So:

• Maximum parallelism for a topic in one group = number of partitions.
• Throughput of a topic into Elasticsearch is bounded by partitions x per-consumer throughput.
• Adding Logstash instances beyond the partition count does nothing for that topic.

In Logstash, the Kafka input's consumer_threads (default 1) sets how many Kafka consumers one input runs. The total consumer count for a group is the sum across all Logstash instances with the same group_id (default logstash):

input {
  kafka {
    bootstrap_servers => "kafka-01:9093,kafka-02:9093,kafka-03:9093"
    topics            => ["logs-web"]
    group_id          => "es-logs-web"
    consumer_threads  => 4
    codec             => "json"
    decorate_events   => "basic"
    security_protocol => "SSL"
  }
}

Three Logstash instances with consumer_threads => 4 give 12 consumers. A topic with 12 partitions gives each one partition; a topic with 24 gives each two. A topic with 8 leaves four consumers idle.

A practical rule of thumb is to create more partitions than you need today, as a multiple of your expected consumer counts (12, 24 and 48 divide evenly across 2, 3, 4 and 6 consumers), because adding partitions later changes key-to-partition mapping for keyed topics and reshuffles assignment. Do not go to extremes: every partition has a cost on the brokers and lengthens rebalances.

Use a distinct group_id per logical consumer. If two different Logstash pipelines share a group id across different topic lists, they join the same group and rebalances interfere with each other. A second cluster consuming the same topic for a migration needs its own group id, so it gets its own copy of every event.

decorate_events adds Kafka metadata under [@metadata][kafka], including topic, partition and offset. That metadata is what you use for routing and for deterministic ids below. Because it lives under @metadata, it is not sent to Elasticsearch unless you copy it into a field.

How backpressure travels

Backpressure is where Kafka architectures are most often misunderstood. Follow a slow cluster step by step:

1. Elasticsearch write thread pools fill and bulk requests start getting 429 rejections, or bulk latency rises sharply. See /guides/elasticsearch-thread-pools-rejections/.
2. The Logstash Elasticsearch output retries rejected items with exponential backoff, starting at retry_initial_interval (default 2 seconds) and growing up to retry_max_interval (default 64 seconds). Workers block while they retry.
3. With workers blocked, the Logstash queue between inputs and filters fills. With the default in-memory queue that happens almost immediately; with a persisted queue it happens when queue.max_bytes is reached.
4. Once the queue is full, the Kafka input cannot hand off events and stops consuming.
5. Kafka keeps accepting writes from producers. Consumer lag grows.
6. Producers are unaffected until Kafka itself is out of capacity or the retention window is exceeded.

This is the decoupling working as designed. Two things can break it.

Consumer group churn. A Kafka consumer must call poll within max.poll.interval.ms (default 300,000 ms, five minutes) or the broker considers it failed and rebalances its partitions to other consumers. If a Logstash input blocks for longer than that because the pipeline is stalled, its consumers are kicked out, the group rebalances, other consumers pick up the partitions, they also stall, and the group churns. Symptom: repeated rebalance messages in Logstash logs and lag that does not drain even after Elasticsearch recovers. Confirm by checking the consumer group's members repeatedly. Mitigations are to keep batches small enough to finish quickly, size the persisted queue so the input can keep accepting for a while, and fix the slow cluster rather than tuning the timeout upward indefinitely.

Lag that outlives retention. If lag grows past Kafka's retention, the oldest unconsumed messages are deleted and lost. Retention is the size of your outage budget. Broker default retention is seven days (log.retention.hours 168) unless overridden per topic with retention.ms. Set retention per topic from how long a cluster outage you want to survive plus replay needs, then check that disks can hold it.

The drain rate after an outage is also bounded. If the cluster can index 1.3 times your normal rate, a 4 hour backlog takes 4 / 0.3 = 13.3 hours to clear while new data keeps arriving. During that time your dashboards show data hours behind. Plan for catch-up capacity explicitly, as in the sizing section.

Delivery guarantees and the exactly-once myth

Kafka supports exactly-once semantics for transactions within Kafka: read from a topic, process, write to another topic and commit offsets atomically. Elasticsearch is not a participant in a Kafka transaction. The end-to-end guarantee from Kafka into Elasticsearch is at-least-once at best, and that depends on configuration.

Where events can be lost

Since Kafka 3.0 the Java producer defaults to acks=all with idempotence enabled, but many shippers are not the Java producer and set their own defaults. Check each shipper's Kafka output configuration explicitly rather than assuming.

The Logstash Kafka input commits offsets automatically by default (enable_auto_commit is true). Committed offsets mean "this consumer has read up to here", not "this has been indexed". With an in-memory queue, a crash after a commit loses whatever was in memory. A persisted queue closes most of that gap because events are written to disk before they are acknowledged to the input. The dead letter queue captures events Elasticsearch rejected with errors that will not succeed on retry, such as mapping conflicts. Both are covered in /guides/logstash-pipeline-architecture/.

Where events are duplicated

At-least-once means duplicates. They come from Logstash retrying a bulk request whose items actually succeeded, from a consumer restart re-reading from the last committed offset, and from producer retries on shippers without idempotence.

You can make most duplicates harmless with a deterministic document id: the same event always gets the same _id, so a second write is rejected or overwrites instead of creating a second document.

filter {
  fingerprint {
    source              => ["[@metadata][kafka][topic]", "[@metadata][kafka][partition]", "[@metadata][kafka][offset]"]
    concatenate_sources => true
    method              => "SHA256"
    target              => "[@metadata][doc_id]"
  }
}
output {
  elasticsearch {
    hosts       => ["https://es-coord-01:9200"]
    data_stream => "true"
    document_id => "%{[@metadata][doc_id]}"
  }
}

Topic, partition and offset uniquely identify a Kafka record, so this id deduplicates consumer-side replays. It does not deduplicate a record that a non-idempotent producer wrote twice, because those are two offsets. For that, fingerprint the event content instead (for example the original message plus host plus timestamp).

Data streams only accept the create operation, so a second write with the same id fails with a 409 conflict, which the Logstash output does not retry. That is the dedup working.

The costs of deterministic ids are real and you should accept them deliberately:

• Indexing throughput drops. With auto-generated ids Elasticsearch can skip checking whether the id already exists. With your own ids, every write must look up the id first. See /guides/elasticsearch-indexing-performance/.
• Dedup is per backing index. A data stream writes to its current write index. If the original was written before a rollover and the duplicate arrives after, they land in different backing indices and both survive. Dedup holds within a rollover period, which covers the common case of short-term retries, not a replay a week later.

The honest summary: use auto-generated ids if your consumers of the data tolerate occasional duplicates (most log analytics does), and deterministic ids where counts matter (billing events, audit trails), knowing the throughput cost and the rollover caveat.

Choosing the consumer: Logstash or Elastic Agent

For a high-volume, multi-topic buffer with routing and replay requirements, Logstash is the safer consumer today. For a simpler path where integrations already do the parsing and Fleet is your management plane, an Agent-based consumer can remove a tier. See /guides/elastic-agent-fleet-vs-beats/ for the Fleet trade-offs. On the producer side, Elastic Agent supports a Kafka output in recent 8.x releases, and Filebeat has long had one.

Sizing worked through

Assumptions for this example, stated so you can substitute your own:

• Average ingest 80,000 events per second, peak 200,000 events per second for up to 2 hours.
• Average event size on the wire 800 bytes.
• Kafka replication factor 3, producer compression ratio of roughly 4 to 1 on log text (a rough assumption; measure your own with your codec).
• Retention target: survive a 48 hour cluster outage plus 24 hours of replay margin, so 72 hours.
• Measured throughput of one Logstash consumer thread, end to end into Elasticsearch, 6,000 events per second (measure this in a load test; it depends on filters, output batch size and cluster speed).
• Catch-up target: the cluster and consumers can run at 1.5 times average for draining.

Kafka storage

1. Average bytes per second: 80,000 x 800 = 64,000,000 B/s = 64 MB/s.
2. Per day: 64 MB/s x 86,400 s = 5,529,600 MB, about 5.53 TB/day uncompressed.
3. 72 hours of retention: 5.53 x 3 = 16.6 TB.
4. After 4:1 compression: 16.6 / 4 = 4.15 TB.
5. With replication factor 3: 4.15 x 3 = 12.4 TB of broker disk.
6. Keep brokers below about 70% disk (a rule of thumb, to leave room for bursts and reassignments): 12.4 / 0.7 = 17.8 TB total. Across 6 brokers, that is about 3 TB of usable disk each.

Partitions and consumers

1. Peak rate 200,000 EPS / 6,000 EPS per consumer = 33.3 consumers to keep up at peak without lag.
2. If you choose to absorb the peak as lag instead, size for catch-up: 80,000 x 1.5 = 120,000 EPS / 6,000 = 20 consumers.
3. Choose partitions so the consumer count divides evenly and leaves room to grow: 36 or 48 partitions. With 48 partitions, 4 Logstash instances at consumer_threads => 12 give 48 consumers, enough for peak.

Lag budget

If you size consumers for 120,000 EPS and a 2 hour peak of 200,000 EPS arrives:

1. Excess rate during the peak: 200,000 - 120,000 = 80,000 EPS.
2. Backlog after 2 hours: 80,000 x 7,200 = 576 million events.
3. After the peak, normal input is 80,000 and drain capacity is 120,000, so the backlog drains at 40,000 EPS.
4. Drain time: 576,000,000 / 40,000 = 14,400 s = 4 hours.

The backlog is largest, 576 million events, at the moment the peak ends. Freshness keeps getting worse for a while after that: at the end of the peak the consumers are reading events from about 48 minutes earlier (they have consumed 120,000 x 7,200 = 864 million events, which the peak produced in 864,000,000 / 200,000 = 4,320 s, so 7,200 - 4,320 = 2,880 s behind), and while they work through the remaining peak-rate events at 120,000 EPS the gap widens to about 80 minutes (576,000,000 / 120,000 = 4,800 s). From there it closes and data is back to real time about 4 hours after the peak ends. If that is not acceptable for alerting, size consumers for peak, or put alerting-critical datasets on their own topic with consumers sized for their peak.

Elasticsearch must keep up too

Consumers are only as fast as the cluster they write to. At 120,000 EPS catch-up the cluster needs indexing capacity for 1.5 times normal, on the hot tier, while also serving searches. That sizing is covered in /guides/elasticsearch-capacity-planning-logging/. A Kafka buffer is an excuse to size the cluster for average plus catch-up rather than for peak, which is often the real financial argument for Kafka.

Failure modes and anti-patterns

More consumers than partitions. Symptom: adding Logstash instances does not reduce lag. Confirm with the consumer group description, which shows partitions per consumer and idle members. Fix by increasing partitions, or accept the ceiling.

One hot partition. Symptom: total lag is mostly on one or two partitions. Cause: keyed messages with a skewed key. Confirm with per-partition lag. Fix with a null key or a key with more spread.

Rebalance storms. Symptom: frequent rebalances and lag that will not drain. Cause: consumers exceeding max.poll.interval.ms while the pipeline is blocked, or consumers flapping because of restarts. Fix the stall, size the persisted queue, avoid deploying all Logstash instances at once.

Shared group id across unrelated pipelines. Symptom: pipelines rebalance whenever an unrelated one restarts. Fix with a unique group_id per pipeline.

Retention shorter than the worst outage. Symptom: data missing for the start of an outage window after the cluster recovers. Fix with per-topic retention.ms sized from your outage budget, and an alert on lag in time, not just in messages.

Assuming exactly-once. Symptom: duplicate documents in counts after a consumer restart. Fix with deterministic ids where it matters, and stop promising exactly-once in design documents.

Treating Kafka as a fix for a slow cluster. Symptom: lag grows every day and never fully drains. The cluster's sustained capacity is below the average input rate. A buffer only moves the problem in time. Fix the cluster.

Parsing in the consumer and the cluster. Symptom: pipeline failures on fields already transformed. Fix by owning each transformation in one place.

Runbook: lag is growing

1. Measure lag per partition for the group:

bash kafka-consumer-groups.sh --bootstrap-server kafka-01:9093 \ --command-config client.properties \ --describe --group es-logs-web

Read CURRENT-OFFSET, LOG-END-OFFSET, LAG and CONSUMER-ID for every partition. Partitions with no consumer id are unassigned. 2. If lag is concentrated on a few partitions, it is skew. If it is spread evenly, it is capacity. 3. Check the consumers: Logstash node stats at GET _node/stats/pipelines on port 9600 show events in and out per pipeline and per plugin, and the queue fill level. An input that has stopped emitting with a full queue means the bottleneck is downstream. 4. Check the cluster: GET _cluster/health, then GET _cat/thread_pool/write?v&h=node_name,active,queue,rejected. Rising rejections mean the cluster is the bottleneck. 5. Check for rebalance churn in Logstash logs. If present, stabilize before scaling. 6. Estimate time to retention: (retention window) minus (age of the oldest unconsumed message). If it is less than the drain time, escalate now. 7. Scale the bottleneck: more consumers (up to partitions) if consumers are the limit, cluster capacity if the cluster is the limit. Do not add consumers when the cluster is rejecting; it makes rejections worse. 8. After recovery, confirm lag reaches zero and stays there across a daily peak.

Key takeaways

• Kafka turns cluster outages and bursts into lag instead of loss, and gives you replay and fan-out. It costs a distributed system.
• Within a consumer group, one partition maps to one consumer. Partitions cap parallelism; size them as a multiple of expected consumer counts.
• Backpressure travels from Elasticsearch through Logstash's queue into Kafka lag; watch max.poll.interval.ms to avoid rebalance storms.
• Delivery into Elasticsearch is at-least-once at best. Use a persisted queue, a dead letter queue, and deterministic ids where duplicates matter.
• Deterministic ids cost indexing throughput and only dedupe within one backing index.
• Size retention for your worst outage, consumers for catch-up, and the cluster for average plus drain.

How Sparced helps

When consumer lag grows, the first question is whether Elasticsearch is the bottleneck, and that is what Sparced answers from its single read-only pass: cluster health, write thread pool queues and rejections per node, heap and circuit breaker pressure, and disk watermark state, each with the nodes and indices named. If the cluster is red or yellow, it runs _cluster/allocation/explain for each stuck shard so you can see why a write index is not accepting data. It also reads ILM state, which catches a stuck rollover that leaves consumers writing into an ever-growing index. It does not read Kafka or Logstash, so consumer lag and pipeline stats remain your checks.