Monitoring Elasticsearch in Production: Metrics, Alerts, Paging

Most Elasticsearch incidents announce themselves hours or days in advance: disk creeping towards the high watermark, heap floors rising after each garbage collection, a growing count of write rejections, an ILM policy that stopped rolling over last Tuesday. Teams get paged at 3 a.m. not because the signals were absent but because nobody was collecting them, the monitoring lived on the cluster that went down, or every signal paged with the same urgency so the important one was ignored. The answer is a small set of metrics with clear thresholds, collected by Metricbeat or Elastic Agent into a separate monitoring cluster, and split into two classes: what threatens availability or data right now, which pages, and what will become a problem, which opens a ticket. This article gives you that set, the APIs behind each metric, the thresholds and the reasoning for them.

The model: what you are actually watching

An Elasticsearch cluster has three layers of state, and each fails differently.

1. Cluster coordination. One elected master holds cluster state: which nodes exist, which indices and shards exist, where every shard copy lives. If the master is unstable, everything that changes cluster state stalls: index creation, rollover, mapping updates, shard allocation. Signals: master changes, pending cluster tasks, node joins and leaves.
2. Shard availability. Every primary must be assigned for its index to be fully writable and searchable; every replica must be assigned for redundancy. Signals: cluster health status, unassigned shards and the reasons the allocation deciders give for them.
3. Node resources. Each node has a finite heap, CPU, disk and set of thread pools. Pressure on any one of them degrades every shard that node holds. Signals: heap after GC, CPU, disk used against watermarks, thread pool queues and rejections, circuit breaker trips.

On top of that sit the workload signals that users feel: indexing rate and latency, search rate and latency, and the background machinery that keeps data flowing (ILM, snapshots, ingest pipelines).

A good monitoring setup covers all four. A common gap is having resource graphs and no availability or background-machinery alerts, so a stuck ILM policy grows one index to several terabytes before anyone notices.

Collecting the data: Stack Monitoring done properly

Metricbeat or Elastic Agent, not internal collection

Elasticsearch used to collect its own monitoring data and ship it to the monitoring indices ("internal collection" or "legacy collection"). That method has been deprecated since 7.16. The supported methods in 8.x and 9.x are:

• Metricbeat with the elasticsearch module in Stack Monitoring mode.
• Elastic Agent with the Elasticsearch integration, managed through Fleet.

A minimal Metricbeat module configuration:

# modules.d/elasticsearch-xpack.yml
- module: elasticsearch
  xpack.enabled: true
  period: 10s
  hosts: ["https://es-node-01:9200"]
  scope: node
  username: "remote_monitoring_user"
  password: "${ES_MON_PASSWORD}"
  ssl.certificate_authorities: ["/etc/metricbeat/ca.crt"]

xpack.enabled: true makes the module collect the metric sets that Stack Monitoring in Kibana expects. scope: node (the default) means each Metricbeat collects from the node it points at, so you run one Metricbeat per Elasticsearch node, usually on the same host. scope: cluster lets one Metricbeat collect from the whole cluster through one endpoint, which is simpler to deploy but makes that one Metricbeat a single point of failure for monitoring. The built-in remote_monitoring_user and the remote_monitoring_collector role exist for this purpose; use them or an equivalent least-privilege user.

Collect the logs too. Elasticsearch server logs, GC logs, slow logs and deprecation logs explain what the metrics only show. Filebeat's Elasticsearch module or the Agent integration ships them in a form Stack Monitoring can link to.

A separate monitoring cluster

Elastic's guidance for production is to send monitoring data to a dedicated monitoring cluster rather than to the cluster being monitored. The reasons are practical:

• When the production cluster is red, overloaded or down, the monitoring data you need to diagnose it is still available.
• Monitoring writes do not add load to a cluster that is already struggling.
• One monitoring cluster can watch several production clusters, so you see them side by side.
• Retention of monitoring data is decided separately from production retention.

Point a Kibana at the monitoring cluster for the Stack Monitoring UI. If you use a single Kibana that also serves production users, monitoring.ui.elasticsearch.hosts lets the Stack Monitoring app read from the monitoring cluster while the rest of Kibana talks to production.

The monitoring cluster should be boring: small, stable, on separate hardware or a separate cloud deployment, and with its own snapshot policy. It does not need to be highly available in the same way production does, but if it goes down silently you are blind, so watch it from outside (see the dead man's switch below).

Sizing the monitoring cluster

Monitoring volume grows with the number of nodes, indices and shards, and with the collection period. A worked estimate with stated assumptions:

• Production: 30 nodes, 600 indices, 4,000 shards (primaries and replicas).
• Collection period: 10 seconds, so 86,400 / 10 = 8,640 collections per day.
• Roughly one document per node, per index and per shard per collection, plus a small number of cluster-level documents. This is an approximation of how the module's metric sets behave; measure your actual daily document count after a day.
• Average document size on disk around 1 KB including overhead (an assumption; check with _cat/indices on the monitoring cluster).

Step by step:

1. Documents per collection: 30 + 600 + 4,000 = 4,630.
2. Per day: 4,630 x 8,640 = 40.0 million documents.
3. At about 1 KB each: about 40 GB per day of primary data.
4. With one replica: about 80 GB per day.
5. For 7 days of retention: about 560 GB; for 30 days: about 2.4 TB.

Shard-level documents dominate. If that volume is a problem, the levers are a longer period for the shard-level data, shorter retention, or fewer shards in production, which is usually the better fix anyway (see /guides/elasticsearch-shard-sizing/).

The metrics that matter

You can graph hundreds of metrics. These are the ones that predict or describe incidents, the API each comes from, and what to look for.

Availability and coordination

GET _cluster/health

{
  "cluster_name": "prod-logs",
  "status": "yellow",
  "number_of_nodes": 30,
  "number_of_data_nodes": 24,
  "active_primary_shards": 2000,
  "active_shards": 3992,
  "relocating_shards": 2,
  "initializing_shards": 0,
  "unassigned_shards": 8,
  "delayed_unassigned_shards": 8,
  "number_of_pending_tasks": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 99.8
}

• status: red means at least one primary is unassigned; yellow means at least one replica is. Red is always a page. Yellow is a page only if it persists and the replicas are not delayed by design.
• unassigned_shards and delayed_unassigned_shards: delayed shards are waiting out index.unassigned.node_left.delayed_timeout (default 1 minute) after a node left. They are expected during a restart.
• number_of_nodes: a drop you did not plan is a page.
• number_of_pending_tasks and task_max_waiting_in_queue_millis: a master that cannot keep up. Seconds of queue wait means cluster state changes are backing up.

For why shards are unassigned, _cluster/allocation/explain gives the decider's answer. See /guides/elasticsearch-unassigned-shards/.

Since 8.x the health report API summarizes many of these checks with a cause and suggested actions:

GET _health_report

It returns an overall status and per-indicator results, with indicators such as master_is_stable, shards_availability, disk, ilm, slm and shards_capacity. Each non-green indicator carries a symptom, the impacted resources and a diagnosis. It is a good input for alerting because the cluster itself does the interpretation, and it is the first call to make during triage.

Node resources

GET _cat/nodes?v&h=name,node.role,heap.percent,ram.percent,cpu,load_1m,disk.used_percent&s=heap.percent:desc

GET _nodes/stats/jvm,os,fs,thread_pool,breaker?filter_path=nodes.*.name,nodes.*.jvm.mem.heap_used_percent,nodes.*.jvm.gc.collectors.old,nodes.*.thread_pool.write,nodes.*.thread_pool.search,nodes.*.breakers.parent

Heap. jvm.mem.heap_used_percent is a sawtooth: it rises as objects are allocated and drops after collection. A momentary 90% is normal. What matters is the floor after collections and how long the heap stays high. A rising floor, or heap above 85% for many minutes, means the live data set no longer fits comfortably and the node is heading for long pauses or circuit breaker trips. jvm.gc.collectors.old.collection_count and collection_time_in_millis are cumulative; their rate tells you whether old-generation collection is becoming frequent or slow. Background in /guides/elasticsearch-jvm-heap-circuit-breakers/.

Circuit breakers. breakers.parent.tripped (and the per-breaker tripped counts) are cumulative. Any increase means requests are being refused to protect the heap. A trip on the parent breaker is a strong signal of memory pressure.

Disk. disk.used_percent against the watermarks. The defaults are low 85%, high 90% and flood stage 95%. At low, Elasticsearch stops allocating replicas and relocating shards to the node (primaries of newly created indices can still land there); at high, Elasticsearch moves shards away; at flood stage, indices with a shard on that node get a read_only_allow_delete block and writes fail. Large disks use absolute headroom caps in recent versions, so check the effective thresholds. See /guides/elasticsearch-disk-watermarks/.

Thread pools. For write and search, watch queue and the rate of rejected (cumulative). A non-zero queue is normal under load; sustained rejections are clients being told to back off. See /guides/elasticsearch-thread-pools-rejections/.

CPU and load. Sustained high CPU on hot nodes is often fine during heavy indexing; correlate with latency before alerting. CPU that is high on one node only usually means a hot shard or a skewed allocation.

Workload

From _nodes/stats/indices or index stats, derive rates by differencing two samples:

• Indexing rate: delta of indexing.index_total over time.
• Indexing latency: delta of indexing.index_time_in_millis / delta of indexing.index_total.
• Search rate: delta of search.query_total.
• Query latency: delta of search.query_time_in_millis / delta of search.query_total.

These are averages over all shards on a node, which hide tail latency. For user-facing search, measure latency at the client or the application as well, and use slow logs to catch outliers. See /guides/elasticsearch-search-latency/.

Background machinery

GET _ilm/explain/logs-*?only_errors=true

Any index in an error step is a ticket at least; a write index whose rollover is failing is a page, because it will grow without bound. See /guides/elasticsearch-ilm-troubleshooting/.

GET _slm/stats

GET _slm/policy/nightly-snapshots

The policy response includes the last success and last failure. A failed snapshot is a ticket; no successful snapshot for longer than your recovery point objective is a page. See /guides/elasticsearch-snapshot-restore-dr/.

Shard count. The total shard count against the limit cluster.max_shards_per_node (default 1,000 per non-frozen data node) multiplied by data nodes. When the limit is reached, index creation fails, which in a logging cluster means rollover fails. The shards_capacity indicator in the health report tracks this for you.

Thresholds

These thresholds are starting points. Where a value is an Elasticsearch default it is labeled as such; the rest are common rules of thumb that you should adjust to your cluster's baseline.

The duration terms matter as much as the thresholds. Heap and CPU are spiky by nature; alerting on instantaneous values creates noise that teaches people to ignore alerts.

Page or ticket: the decision rule

Use one rule to sort every alert: page when data or availability is at risk now and a human can do something about it in the next hour; otherwise ticket.

Two practices make this work. First, suppress during maintenance: a rolling restart produces exactly the yellow status and node departures that page in normal times. Second, route tickets somewhere they are actually reviewed, weekly at least, because ticket-class alerts are where incidents start.

Alerting tools and the dead man's switch

Stack Monitoring in Kibana ships with built-in rules for common conditions, including CPU usage, disk usage, JVM memory usage, missing monitoring data, thread pool rejections, cluster health, node changes and license expiration. They are a reasonable baseline. Review their thresholds and durations against the table above and adjust them rather than accepting all defaults.

For conditions the built-in rules do not cover (ILM errors, snapshot age, workload latency), write Kibana rules against the monitoring data, or poll the APIs from your existing monitoring system. If your organization already pages through another platform, a small exporter that polls _cluster/health, _health_report, _nodes/stats and _ilm/explain and forwards the results is often more reliable than duplicating routing rules in two systems.

Finally, monitor the monitoring. Every alerting system fails silently at some point. Add a check from outside the monitoring cluster that fires when monitoring data stops arriving, and a periodic test alert that someone must acknowledge. The missing monitoring data rule in Stack Monitoring covers the case where one production node stops reporting; it cannot cover the case where the monitoring cluster or its Kibana is itself down.

Failure modes and anti-patterns

Monitoring on the production cluster. Symptom: during an outage the Stack Monitoring UI is empty or times out. Fix with a separate monitoring cluster.

Alerting on instantaneous heap. Symptom: heap alerts every few minutes that clear themselves, followed by an ignored real one. Fix with sustained-duration conditions and post-GC heap.

Watching cluster health only. Symptom: green cluster, users complaining. Causes include search rejections, slow queries, ILM stuck, ingest pipeline failures. Fix with workload and background-machinery alerts.

Cumulative counters graphed raw. Symptom: a rejection counter graph that only ever goes up, which nobody can read. Fix by alerting on the rate of change.

No maintenance suppression. Symptom: every rolling restart pages the on-call. Fix with scheduled silences tied to change windows.

One Metricbeat in cluster scope for everything. Symptom: monitoring gap whenever that host restarts. Fix with node-scope Metricbeat per node, or redundant collectors.

Thresholds copied, not baselined. Symptom: CPU alerts on hot nodes during normal ingest. Fix by measuring two weeks of baseline and setting thresholds from it.

Monitoring retention too short. Symptom: you cannot compare this week's heap floor with last month's. Fix by keeping at least 30 days of downsampled or lower-resolution data for trends, even if high-resolution data is kept for a week.

Runbook: the first ten minutes of any alert

1. GET _health_report. Read every non-green indicator's diagnosis and impacted resources.
2. GET _cluster/health. Note status, node count, unassigned and pending tasks.
3. If red or yellow without an explanation: GET _cluster/allocation/explain for one unassigned shard, then check GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason&s=state for the scope.
4. GET _cat/nodes?v&h=name,node.role,heap.percent,cpu,load_1m,disk.used_percent&s=heap.percent:desc. Find the node that stands out.
5. GET _cat/allocation?v. Check disk per node against the watermarks.
6. GET _cat/thread_pool/write,search?v&h=node_name,name,active,queue,rejected. Find rejections and which nodes carry them.
7. GET _nodes/stats/breaker?filter_path=nodes.*.name,nodes.*.breakers.parent. Check for trips.
8. GET _ilm/explain/*?only_errors=true if the alert involves growth or rollover.
9. Read the master's and the outlying node's logs for the alert window.
10. Decide: mitigate (stop a heavy query, pause ingest, add disk), escalate, or hand to a ticket with the evidence collected above. Use /guides/elasticsearch-red-cluster-runbook/ if the cluster is red.

Key takeaways

• Collect with Metricbeat or Elastic Agent into a separate monitoring cluster; internal collection is deprecated.
• Watch four layers: coordination, shard availability, node resources and workload, plus ILM, snapshots and shard count.
• Alert on sustained conditions and rates, not instantaneous values or raw cumulative counters.
• Page when data or availability is at risk now and a human can act; ticket everything with lead time, and review tickets.
• Use _health_report as the first triage call and a good alerting input.
• Monitor the monitoring with an external dead man's switch.

How Sparced helps

Sparced is a point-in-time diagnostic rather than a continuous monitor, and it fits the runbook above: in one read-only pass it reads cluster health, nodes, shards, allocation, thread pools, circuit breakers, ILM, snapshots and security across 81 endpoints and reports each problem with the specific nodes, indices or shards involved. When the cluster is red or yellow, or a node is past 85% heap, it makes up to 9 further calls, including _cluster/allocation/explain for each stuck shard, so the first ten minutes of triage arrive already collected. Each finding comes with a fix command with names filled in and a check to confirm it worked. It installs no agent, writes nothing and does not replace alerting: it is what you run when an alert fires, or before a change, to see everything at once.