Ingest Pipelines vs Logstash: Where to Parse Your Logs

Every log line that reaches Elasticsearch has to be turned from a string into fields somewhere, and you have two serious places to do it: an ingest pipeline that runs inside the cluster, or a Logstash filter stage that runs before the cluster. The short answer is that ingest pipelines should be your default for parsing, enrichment and normalization that can be expressed one document at a time, and Logstash earns its place when you need buffering, fan-out to several destinations, lookups against external systems, or cross-event logic. The rest of this article is the model behind that answer, the cost of the processors that dominate CPU, how to make failures visible instead of silent, and how to size the nodes that do the work.

The two execution models

Both options do the same job, but they run in very different places and they fail in very different ways.

Ingest pipelines run on the write path

An ingest pipeline is a named, ordered list of processors stored in cluster state. When an index or bulk request names a pipeline, or the target index has one configured as a default, the document is transformed before it is indexed. The sequence for a bulk request is:

1. The request arrives at a node, which acts as the coordinating node.
2. If that node does not have the ingest role, it forwards the request to a node that does.
3. The ingest node runs every document through its pipeline, in memory, one document at a time.
4. The transformed documents are then routed to the primary shards that own them, exactly as an ordinary bulk request would be.

Two consequences follow. First, ingest work uses the same CPU and the same heap as indexing on those nodes. Elastic's thread pool documentation lists ingest processing alongside write operations on the write pool, so heavy pipelines show up as write queueing and, eventually, write rejections. Second, there is no buffer. If the ingest nodes are saturated, the client sees 429 responses and must retry. Backpressure goes straight back to the shipper.

By default every node has the ingest role. In small clusters that is fine. In larger clusters, a common pattern is to remove ingest from master-eligible nodes and to decide deliberately whether parsing runs on the data nodes or on a dedicated set of ingest nodes. See /guides/elasticsearch-cluster-topology/ for the role layout.

Logstash runs before the cluster

Logstash is a separate JVM process with an input stage, a filter stage and an output stage. Events are pulled from inputs into a queue, then pipeline workers take batches from the queue and run them through filters and outputs. The defaults that matter:

Because Logstash has its own queue, it can absorb a burst, survive a short Elasticsearch outage (with a persisted queue), and send one event to several outputs. Because it runs outside the cluster, its CPU cost is not competing with indexing and search. The price is another tier to deploy, patch, monitor and scale. The internals of workers, persistent queues and pipeline-to-pipeline patterns are covered in /guides/logstash-pipeline-architecture/.

What is genuinely different

The row that settles most decisions is the integrations row. Elastic Agent integrations and Filebeat modules ship their parsing as ingest pipelines. If your sources are covered by an integration, the parsing already exists as an ingest pipeline and rewriting it in Logstash is work with no payoff.

Defining, attaching and chaining pipelines

A pipeline is created with a single request. This one parses an nginx-style access line, sets the timestamp and removes the raw field only after success:

PUT _ingest/pipeline/web-access
{
  "description": "Parse web access logs",
  "processors": [
    {
      "dissect": {
        "field": "message",
        "pattern": "%{source.ip} - %{user.name} [%{_tmp.ts}] \"%{http.request.method} %{url.original} HTTP/%{http.version}\" %{http.response.status_code} %{http.response.body.bytes}",
        "tag": "dissect-access"
      }
    },
    {
      "date": {
        "field": "_tmp.ts",
        "formats": ["dd/MMM/yyyy:HH:mm:ss Z"],
        "target_field": "@timestamp",
        "tag": "date-access"
      }
    },
    {
      "convert": {
        "field": "http.response.status_code",
        "type": "long",
        "ignore_missing": true
      }
    },
    {
      "remove": {
        "field": ["_tmp", "message"],
        "ignore_missing": true
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "field": "event.kind",
        "value": "pipeline_error"
      }
    },
    {
      "set": {
        "field": "error.message",
        "value": "{{ _ingest.on_failure_message }}"
      }
    }
  ]
}

Every processor accepts a common set of options: if (a Painless condition), tag, description, ignore_failure and its own on_failure list. Tags matter more than people expect, because they are what appears in node stats and in failure metadata. Untagged processors are anonymous in both places.

Attaching a pipeline to an index or data stream

There are three ways a document gets a pipeline:

1. Per request: POST my-index/_bulk?pipeline=web-access, or a pipeline field on each bulk action.
2. index.default_pipeline: an index setting that applies when the request names no pipeline. A request-level pipeline replaces it. The special value _none disables it for a request.
3. index.final_pipeline: an index setting that always runs after the request or default pipeline, and cannot be bypassed by the client.

For data streams, set these in the index template so every new backing index inherits them on rollover:

PUT _index_template/logs-web
{
  "index_patterns": ["logs-web-*"],
  "data_stream": {},
  "priority": 200,
  "template": {
    "settings": {
      "index.default_pipeline": "web-access",
      "index.final_pipeline": "logs-governance"
    }
  }
}

The distinction between default and final is an architectural lever. Put parsing in default_pipeline, which a client can legitimately override (a backfill job may send already-parsed documents with pipeline=_none). Put governance in final_pipeline: field removal for compliance, setting event.ingested, dropping debug fields, anything that must happen whatever the client does. Template changes only apply to backing indices created after the change, so after editing a template you either roll the data stream over or update the setting on the current write index. Template mechanics are in /guides/elasticsearch-index-templates-data-streams/.

Chaining and routing

The pipeline processor calls another pipeline by name, which lets you build a small library: a shared common-ecs pipeline called from each source pipeline, or a router that dispatches on a field:

{
  "processors": [
    { "pipeline": { "name": "parse-nginx", "if": "ctx.service?.type == 'nginx'" } },
    { "pipeline": { "name": "parse-haproxy", "if": "ctx.service?.type == 'haproxy'" } }
  ]
}

Elastic Agent integrations use the same mechanism to call a user-owned @custom pipeline, for example logs-nginx.access@custom. That is the supported hook for adding your own processors to an integration without editing the managed pipeline, which is overwritten when the integration is upgraded.

In recent 8.x releases there is also a reroute processor that changes the target data stream of a document from inside a pipeline, which is how a single generic stream can be split by dataset or namespace on the cluster side.

Grok versus dissect: where the CPU goes

On most logging clusters the parse step is the single largest cost in a pipeline, and the choice between grok and dissect dominates it.

Dissect splits a string on literal delimiters. It walks the input once, has no backtracking and no regular expressions, and its cost is roughly linear in the length of the line. Its syntax is small: %{field} captures, %{?name} skips, %{+field} appends to an earlier capture, and the -> modifier after a key absorbs repeated delimiters such as padding spaces. It works when the line has a fixed shape.

Grok compiles named patterns into regular expressions and matches them. It handles optional parts, alternations and variable shapes, and the cost of that flexibility is backtracking. A pattern that fails to match may try many alternatives before giving up, and a badly anchored pattern such as one that starts with %{GREEDYDATA} can be dramatically slower on non-matching lines than on matching ones.

The pattern that works best in practice is a combination: dissect the stable prefix (timestamp, level, logger, thread) into fields, then apply grok only to the remaining free-text part, and only when an if condition says the event is of a type that needs it. That keeps the expensive engine off most events and off most of each line.

Grok guards against runaway expressions in both products. In Elasticsearch a watchdog interrupts grok expressions that run too long; its limit is the node setting ingest.grok.watchdog.max_execution_time, which defaults to one second (the companion ingest.grok.watchdog.interval, also 1s, is deprecated as of 9.4 because it is no longer needed). In Logstash the grok filter has timeout_millis, defaulting to 30,000 ms. If you are hitting either, the fix is the pattern, not the timeout.

Rules that keep grok cheap:

• Anchor patterns with ^ so a non-match fails at the first character, not after scanning the whole line.
• Order alternatives in a patterns list from most to least frequent; grok stops at the first match.
• Avoid leading GREEDYDATA and DATA; prefer specific patterns such as NOTSPACE, INT or IPORHOST.
• Put a cheap if in front: "if": "ctx.message != null && ctx.message.startsWith('ERROR')".

GET _ingest/processor/grok returns the built-in pattern library, which is useful when you want to see what a named pattern actually expands to before relying on it.

The simulate API is your test harness

Never put a pipeline in front of production traffic without running real samples through _simulate. It executes the pipeline against documents you supply and indexes nothing.

POST _ingest/pipeline/web-access/_simulate?verbose=true
{
  "docs": [
    { "_source": { "message": "10.1.2.3 - alice [21/Oct/2026:10:15:02 +0000] \"GET /api/v1/items HTTP/1.1\" 200 5123" } },
    { "_source": { "message": "garbage line that will not match" } }
  ]
}

With verbose=true the response contains one entry per processor per document, so you can see exactly where a document changed or failed:

{
  "docs": [
    {
      "processor_results": [
        { "processor_type": "dissect", "tag": "dissect-access", "status": "success", "doc": { "_source": { "source": { "ip": "10.1.2.3" } } } },
        { "processor_type": "date", "tag": "date-access", "status": "success" }
      ]
    },
    {
      "processor_results": [
        { "processor_type": "dissect", "tag": "dissect-access", "status": "error", "error": { "reason": "Unable to find match for dissect pattern: ... against source: garbage line that will not match" } }
      ]
    }
  ]
}

The fields to read are status (success, error, error_ignored, skipped when an if was false, or dropped), tag, and the doc snapshot after each step. You can also simulate a pipeline definition that is not stored yet by sending "pipeline": { ... } in the body of POST _ingest/pipeline/_simulate, which is how to test a change before you PUT it.

A disciplined workflow: keep a file of real sample lines per source, including the ugly ones (multi-line stack traces, truncated lines, IPv6 addresses, empty user fields), and simulate against all of them in CI before every pipeline change.

Making failures visible with on_failure

The default behavior of an ingest pipeline when a processor throws is to reject the document. In a bulk request that means one item in the response carries an error and the rest succeed. Many shippers log that item error and move on, so a broken pipeline silently loses data while the cluster looks healthy.

You have three tools, and each has a cost:

The pattern that holds up in production is a pipeline-level on_failure that keeps the document and marks it, plus a dashboard or alert on the marker. Inside the handler, these metadata fields describe the failure: _ingest.on_failure_message, _ingest.on_failure_processor_type, _ingest.on_failure_processor_tag and _ingest.on_failure_pipeline.

"on_failure": [
  { "set": { "field": "error.message", "value": "{{ _ingest.on_failure_message }}" } },
  { "set": { "field": "error.type", "value": "{{ _ingest.on_failure_processor_type }}" } },
  { "set": { "field": "event.pipeline_failed_at", "value": "{{ _ingest.on_failure_processor_tag }}" } }
]

With plain indices, a handler that sets _index to a failure index (for example failed-{{ _index }}) is a common way to quarantine bad documents. With data streams, be careful: redirecting to a name that is not a valid target or that collides with a data stream's naming scheme will itself fail. Recent Elasticsearch releases add a failure store for data streams that captures documents rejected by ingest or mapping, which is cleaner than hand-rolled redirection where it is available on your version. Check your version's documentation before relying on it.

Two processors are specifically for flow control: drop discards a document deliberately (for example health-check noise), and fail raises an error with a custom message, which is how you enforce a contract such as "every event must carry service.name".

Enrichment, lookups and what each side can reach

Enrichment is where the two options diverge the most.

Ingest pipelines reach data that already lives in the cluster or ships with it:

• geoip adds location from MaxMind-format databases. Elasticsearch downloads the GeoLite2 databases automatically by default, controlled by ingest.geoip.downloader.enabled. Air-gapped clusters need the databases provided another way, and a pipeline that runs before the databases are available tags documents rather than failing them.
• user_agent parses browser strings.
• enrich joins against a lookup index built from a source index by an enrich policy.

The enrich workflow has a trap. The enrich index is a read-only snapshot of the source index at the time the policy was executed:

PUT _enrich/policy/host-owner
{
  "match": {
    "indices": "cmdb-hosts",
    "match_field": "host.name",
    "enrich_fields": ["owner.team", "owner.email", "host.tier"]
  }
}

POST _enrich/policy/host-owner/_execute

{
  "enrich": {
    "policy_name": "host-owner",
    "field": "host.name",
    "target_field": "cmdb",
    "max_matches": 1,
    "ignore_missing": true
  }
}

Updates to cmdb-hosts are invisible until you execute the policy again. If your CMDB changes hourly, schedule the _execute call hourly, and monitor that it runs. Enrich lookups also consume coordinating resources on the ingest node, and a large enrich index with high lookup rates adds measurable latency per document.

Logstash can reach anything with a network address: a database through jdbc_streaming, a REST service through http, a dictionary file through translate, or Elasticsearch itself through the elasticsearch filter. That flexibility is real, and so is its failure mode: an external call per event couples your ingest rate to the latency and availability of that system. Cache aggressively, and treat any per-event network call as a design smell unless the rate is low.

If your lookups are against slowly changing reference data that you can load into an index, enrich keeps the architecture simpler. If they are against a live system of record, Logstash is the honest choice.

Measuring pipeline cost and sizing ingest nodes

You cannot size what you have not measured, and Elasticsearch measures ingest for you:

GET _nodes/stats/ingest?filter_path=nodes.*.name,nodes.*.ingest.total,nodes.*.ingest.pipelines.web-access

{
  "nodes": {
    "a1b2": {
      "name": "es-ingest-01",
      "ingest": {
        "total": { "count": 918234112, "time_in_millis": 188470211, "current": 37, "failed": 12044 },
        "pipelines": {
          "web-access": {
            "count": 512003311,
            "time_in_millis": 97302112,
            "current": 21,
            "failed": 12011,
            "processors": [
              { "dissect:dissect-access": { "type": "dissect", "stats": { "count": 512003311, "time_in_millis": 21880211, "current": 0, "failed": 9 } } }
            ]
          }
        }
      }
    }
  }
}

The numbers are cumulative since the node started, so take two samples a few minutes apart and difference them. The key derived figure is milliseconds of ingest time per document:

ms_per_doc = delta(time_in_millis) / delta(count)

failed is the count to alert on: a rising failed on a pipeline with no on_failure is documents being rejected. Per-processor stats tell you which processor is expensive. A tagged processor is listed as type:tag (here dissect:dissect-access), an untagged one only by its type, which is why tags matter.

Worked sizing example

Assumptions, stated so you can replace them with your own measurements:

• Peak ingest rate: 60,000 events per second.
• Measured pipeline cost: 0.25 ms of ingest time per event, from the node stats delta above.
• Target: ingest work should use no more than 50% of the cores on the nodes that run it, leaving the rest for indexing, merging and searches (a rule of thumb, not Elastic guidance).
• Node size: 16 cores.

Step by step:

1. Ingest CPU time needed per second: 60,000 x 0.25 ms = 15,000 ms, which is 15 core-seconds per second, so 15 cores fully busy.
2. At a 50% budget, you need 15 / 0.5 = 30 cores of ingest-capable capacity.
3. On 16-core nodes: 30 / 16 = 1.9, so 2 nodes minimum, and 3 to survive the loss of one at peak.

Two cautions. time_in_millis is wall-clock time inside the pipeline, which includes waiting (enrich lookups, for example), so it is an upper bound on CPU. And the measurement changes with the data: a deploy that changes log format can turn cheap dissect matches into expensive grok fall-throughs overnight. Re-measure after any source change.

Dedicated ingest nodes or not

Dedicated ingest nodes are CPU-heavy and storage-light: they hold no shards. Give them modest heap, because ingest is mostly transient per-document memory, and spend the budget on cores.

A decision framework

Use this order of questions:

1. Is the source covered by an Elastic integration or Beats module? Use its ingest pipeline. Add customization in the @custom hook.
2. Do you need a buffer, because the shippers cannot retry or the source is bursty? Put Kafka or a Logstash persisted queue in front. That still does not force parsing into Logstash. See /guides/logging-architecture-kafka-elasticsearch/.
3. Do you need several destinations for the same event (Elasticsearch plus an archive bucket plus a SIEM)? Logstash, or the buffer layer.
4. Do you need lookups against live external systems, or cross-event correlation? Logstash.
5. Is the parse expensive and the cluster's write path already tight? Either move it to Logstash, or add dedicated ingest nodes. Measure which is cheaper to run.
6. Otherwise: ingest pipelines. One less tier, stored with the data, testable with _simulate.

A hybrid is common and sensible: Logstash for transport, buffering and routing, ingest pipelines for parsing. In recent 8.x releases Logstash also has an elastic_integration filter that runs integration ingest pipelines inside Logstash, for teams that need to parse before data leaves a network boundary. If you set the pipeline option on the Logstash Elasticsearch output, Logstash will name an ingest pipeline on every request, and a data stream's default_pipeline will not run for those documents.

Failure modes and anti-patterns

Parsing twice. Logstash groks a field and an ingest pipeline then tries to dissect the same field, which has already been removed or renamed. Symptom: a steady failed count on the pipeline, or error.message values about missing fields. Confirm by simulating with a document shaped exactly like what Logstash emits. Fix by owning each field in one place.

Silent data loss through bulk item errors. Symptom: event counts in Elasticsearch lower than the source, cluster green. Confirm with _nodes/stats/ingest failed counts and by checking the shipper's logs for item-level errors. Fix with a pipeline-level on_failure that keeps and marks documents.

A grok pattern that is fast on good lines and slow on bad ones. Symptom: write latency spikes and write rejections when a new log format appears, with no change in volume. Confirm with per-processor time_in_millis deltas. Fix by anchoring patterns, ordering alternatives, and putting dissect in front.

Client bypassing the default pipeline. Symptom: unparsed documents in a data stream that has a default_pipeline. Cause: the client sends a pipeline parameter (including from a Logstash output) which replaces the default. Fix by moving must-run logic to final_pipeline.

Template edited, nothing changed. Symptom: a new default_pipeline in the template has no effect. Cause: settings apply only to backing indices created after the change. Fix with POST <data-stream>/_rollover or by updating the write index setting directly.

Stale enrichment. Symptom: new hosts appear without owner fields. Cause: the enrich policy has not been executed since the source index changed. Fix by scheduling _execute and alerting when it has not run.

Type conflicts after parse. Symptom: mapper_parsing_exception on bulk items after a pipeline change. Cause: a field that was a string is now a number, or the reverse, against an existing mapping. Fix with explicit convert processors and an explicit mapping in the template. See /guides/elasticsearch-mapping-design/.

Ingest on master-eligible nodes. Symptom: master instability under ingest load. Cause: masters keeping the default ingest role and receiving traffic. Fix: remove the role and point clients at data or ingest nodes only.

Runbook: a pipeline change or an ingest incident

When you are changing a pipeline:

1. Export the current pipeline: GET _ingest/pipeline/<id> and keep it as the rollback.
2. Simulate the new definition with POST _ingest/pipeline/_simulate?verbose=true against the sample corpus, including bad lines.
3. Confirm no error statuses that are not expected, and that field types match the mapping.
4. PUT the pipeline. Pipelines update in place and apply to the next document; there is no rollover needed.
5. Sample _nodes/stats/ingest twice, five minutes apart, and compare failed and ms per document with the baseline.
6. If either regresses, PUT the saved definition back.

When ingest is the suspected cause of a write problem:

1. Check write rejections: GET _cat/thread_pool/write?v&h=node_name,active,queue,rejected.
2. Identify which nodes are doing ingest work: GET _nodes/stats/ingest?filter_path=nodes.*.name,nodes.*.ingest.total.
3. Difference two samples and compute ms per document per pipeline. Find the pipeline with the largest delta(time_in_millis).
4. Within that pipeline, find the processor with the largest delta. Use the tag to locate it.
5. Look at the failed delta. A spike usually means a source changed format.
6. Mitigate: add an if guard, replace grok with dissect for the stable part, or temporarily route the noisy source with drop if it is non-critical and the business agrees.
7. Fix properly, simulate, deploy, and re-measure.

Key takeaways

• Default to ingest pipelines for per-document parsing; use Logstash for buffering, fan-out, live lookups and cross-event logic.
• Dissect is linear and cheap; grok is flexible and can backtrack badly. Dissect the stable prefix and grok only what remains, behind an if.
• Put parsing in index.default_pipeline and must-run governance in index.final_pipeline.
• Simulate every change with verbose=true against real and ugly samples.
• Use a pipeline-level on_failure that keeps and marks documents, and alert on the marker and on node stats failed.
• Size ingest from measured ms per document, not from guesses, and re-measure when sources change.

How Sparced helps

Sparced reads node, node role and thread pool data in its single pass, so write thread pool queueing and rejections on the nodes doing ingest work appear as findings with the node names attached. It also reports heap pressure on those nodes, and when a node is past 85% heap it makes further calls to show the evidence. It does not profile individual pipelines or processors, and it does not change any pipeline: the per-pipeline measurement in this article is still yours to run. What it gives you is a fast answer to whether the write path is under pressure and on which nodes, with a check to confirm when your fix has worked.