Running Kibana at Scale: Instances, Task Manager and Alerting

Kibana looks like a web front end, and at small scale it behaves like one. At scale it is also a distributed job scheduler: alerting rules, reporting jobs, Fleet housekeeping, telemetry and a dozen other background tasks all run inside Kibana processes and coordinate through Elasticsearch. Most Kibana capacity problems are not about page loads; they are about background work falling behind. The answer is to run several identical Kibana instances behind a load balancer, give them the same encryption keys, size them from the Task Manager health API rather than from CPU graphs, separate interactive and background roles when the load warrants it, and treat saved objects as code. This article gives you the model, the settings, the arithmetic and the failure modes.

What Kibana actually is underneath

A Kibana instance is a Node.js server. It holds almost no state of its own. What it does hold lives in Elasticsearch:

• Saved objects: dashboards, visualizations, data views, rules, connectors, spaces, and many more, stored in Kibana system indices. Since 8.x these are split across several .kibana* indices by domain rather than living in one index.
• Task Manager tasks: every scheduled or queued background job is a document in a task index, which instances claim and run.
• Sessions: with security enabled, user sessions are stored server-side in a session index, so any instance can serve any user's request.
• Event log: alerting and action execution history.

That design has a direct consequence. Any number of Kibana instances pointed at the same Elasticsearch cluster, with the same configuration and keys, form a single logical Kibana. There is no Kibana cluster membership protocol, no leader and no quorum. Instances find each other only indirectly, through the documents they read and write.

The other consequence is that every piece of Kibana load is ultimately Elasticsearch load. A dashboard is a set of searches; a rule is a scheduled search; a report is a set of searches plus rendering. When Kibana feels slow, the first question is always whether Kibana is waiting on Elasticsearch.

Running several instances correctly

High availability for Kibana is simply more than one instance behind a load balancer. The mistakes are in the details.

Settings that must be identical on every instance

Each key must be at least 32 characters. If you do not set them, Kibana generates random keys at startup, and for the saved objects key some features, alerting among them, stay unavailable until you set it explicitly. Random keys are tolerable only on a single throwaway instance: sessions are invalidated on every restart and whenever a request lands on an instance with a different key, pending reports fail after a restart, and anything encrypted with a key you no longer have cannot be decrypted. Generate the keys once, store them in the Kibana keystore or your secrets manager, and deploy the same values everywhere.

# kibana.yml, identical on every instance (secrets from the keystore)
server.publicBaseUrl: "https://kibana.example.internal"
elasticsearch.hosts:
  - "https://es-coord-01:9200"
  - "https://es-coord-02:9200"
  - "https://es-coord-03:9200"
elasticsearch.requestTimeout: 30000

Each instance should keep its own data directory, because that is where its instance UUID is stored. A common accident is baking a data directory into a VM image or container volume and cloning it, which gives several instances the same UUID and confuses anything that counts or distinguishes instances, including Task Manager's capacity estimates and Stack Monitoring.

Load balancing

Because sessions live in Elasticsearch, sticky sessions are not required for correctness. They do no harm, and some teams keep them to make per-user debugging easier, but you should be able to remove an instance without logging anyone out.

For health checks, point the load balancer at Kibana's status endpoint:

GET /api/status

The response carries an overall status level and per-plugin statuses. A sensible health check treats the instance as available when the overall level is available, and gives instances generous startup grace, because a Kibana start includes saved object migration checks and plugin setup. Do not use a TCP check only: Kibana can accept connections long before it can serve a page.

On the other side, list several Elasticsearch nodes in elasticsearch.hosts. Kibana spreads requests across them. In larger clusters, point Kibana at coordinating-only nodes so that heavy dashboard aggregations reduce on nodes that hold no data and do not compete with indexing. See /guides/elasticsearch-cluster-topology/ for when coordinating-only nodes are worth it.

elasticsearch.requestTimeout defaults to 30,000 ms. Raising it to make a slow dashboard render is treating the symptom; the dashboard is asking for more than the cluster can answer quickly, which is a search problem. See /guides/elasticsearch-search-latency/.

Task Manager: the part that actually scales

Task Manager is the background job system inside every Kibana instance. Alerting rules, connector actions, reporting, Fleet and many solution features register task types with it. Its model:

1. Tasks are documents in a task index, each with a type, a schedule or a run-at time, and a status.
2. Each Kibana instance polls on an interval and tries to claim tasks that are due, up to its free capacity, using optimistic concurrency so two instances cannot claim the same task.
3. Claimed tasks run inside that Node.js process. When a recurring task finishes, its next run time is written back.

Every instance participates by default. The capacity of the whole system is therefore roughly instances times per-instance capacity, and the two levers are the number of instances and the per-instance worker limit. Historically the per-instance limit was xpack.task_manager.max_workers, default 10, and older releases polled every 3 seconds. Since 8.16 max_workers is deprecated in favor of xpack.task_manager.capacity (default 10, allowed range 5 to 50), and current documentation gives xpack.task_manager.poll_interval a default of 500 ms, so check the Task Manager settings page for your exact version before changing either. In every version the safe default stance is the same: scale out with more instances before you raise the per-instance limit, because each extra concurrent task adds event loop pressure and Elasticsearch load from that one process.

Reading the Task Manager health API

Kibana exposes a health endpoint that is the single best source for Kibana capacity decisions:

GET /api/task_manager/_health

The response has an overall status (OK, warn or error) and a stats object with several sections. The ones that matter:

{
  "status": "OK",
  "stats": {
    "configuration": { "value": { "poll_interval": 500 } },
    "workload": {
      "value": {
        "count": 1842,
        "overdue": 4,
        "task_types": { "alerting:.es-query": { "count": 610 } }
      }
    },
    "runtime": {
      "value": {
        "drift": { "p50": 900, "p90": 2400, "p95": 3100, "p99": 5200 },
        "load": { "p50": 60, "p90": 90, "p95": 100, "p99": 100 },
        "execution": {
          "duration": { "alerting:.es-query": { "p50": 450, "p99": 3900 } }
        }
      }
    },
    "capacity_estimation": { "value": { "observed": {}, "proposed": {} } }
  }
}

Field names vary slightly between versions (the configuration section, for example, reports max_workers on older releases and capacity details on newer ones); the concepts do not.

• drift is how late tasks start compared with when they were due, in milliseconds. This is the headline number. A p99 drift of a few seconds is normal. Drift that keeps growing means tasks are arriving faster than the instances can claim and run them.
• load is the percentage of worker capacity in use at each poll. Sustained p90 load at 100% means every instance is saturated and work queues behind it.
• execution.duration by task type shows which task types are expensive. A single rule type with a p99 of tens of seconds is usually a heavy query behind a rule.
• workload.overdue is the number of tasks past their run time right now.
• capacity_estimation, in versions that include it, compares observed instances with an estimate of how many the current workload needs. Treat it as a hint, not a sizing model.

The health API reports what the instance you asked can see, and some sections are computed per instance. Query every instance directly, not through the load balancer, when you are diagnosing.

Worked capacity example

Assumptions, which you should replace with numbers from the health API:

• 900 alerting rules at a 1 minute interval.
• 200 rules at a 5 minute interval.
• Measured median execution time 1.5 s per rule run, including the Elasticsearch query.
• Per-instance capacity of 10 concurrent tasks (the default; treated here as 10 normal-cost tasks).
• Target sustained load of 60% to absorb bursts (a rule of thumb).

Step by step:

1. Rule runs per minute: 900 + 200 / 5 = 900 + 40 = 940.
2. Worker-seconds per minute: 940 x 1.5 = 1,410.
3. Concurrent workers needed on average: 1,410 / 60 = 23.5.
4. At 60% target load: 23.5 / 0.6 = 39.2 workers.
5. At 10 per instance: 39.2 / 10 = 3.9, so 4 instances, and 5 to survive the loss of one without falling behind.

Two things this arithmetic does not include: other task types (reporting, Fleet, actions triggered by rules) and the Elasticsearch side. 940 rule runs per minute is about 16 searches per second from rules alone, which is a real search workload that the cluster must serve within the rules' timeouts.

Background and UI roles

Since 8.x, Kibana has a node.roles setting that lets you split instances by duty:

# interactive instance, behind the user-facing load balancer
node.roles: ["ui"]

# background instance, not in the user-facing pool
node.roles: ["background_tasks"]

An instance with only ui serves pages and APIs but does not claim background tasks. An instance with only background_tasks runs tasks and is not meant to serve users. The default is both.

The split pays off when you can show interference: user-facing latency rising when rule load peaks, or Node.js event loop delay on instances that also run heavy tasks. Without that evidence, uniform instances are simpler to operate.

Alerting rule load

Rules are the most common reason a Kibana deployment needs to grow, and their cost is mostly on Elasticsearch.

Each rule run is at least one search. The cost of that search depends on the time window, the index pattern, the number of groups for a grouped rule, and whether the query hits cold or frozen data. A rule with a 1 minute interval that looks back 24 hours over logs-* is a 24-hour search, 1,440 times a day.

Rules for keeping rule load sane:

• Match the interval to the decision. A rule that pages someone needs a short interval; a capacity trend rule does not. There is a cluster-wide minimum interval, xpack.alerting.rules.minimumScheduleInterval.value, which defaults to one minute; keep it at least there.
• Keep the look-back window close to the interval, plus a small overlap for ingestion lag. A 1 minute rule rarely needs more than a 5 minute window.
• Narrow the index pattern. A rule over logs-* that only cares about one dataset should target that data stream.
• Watch grouping cardinality. A rule grouped by host.name across 20,000 hosts produces a terms aggregation and can create alerts per group. That cost grows with the group count, not the rule count.
• Avoid rules that search frozen data on a short schedule. Searchable snapshot searches are slow by design. See /guides/elasticsearch-data-tiers-ilm/.

To find the expensive rules, read Task Manager's execution.duration by task type, then the rule's own execution history in the rules UI or the event log, which records durations per run. Sort by duration times frequency, not by duration alone: a 3 second rule every minute costs more than a 30 second rule every hour.

Connector actions are tasks too. A rule that fires to hundreds of groups and sends a webhook per alert can create a burst of action tasks that dwarfs the rule itself. Use summarized actions or throttling where your rule type supports them.

Reporting

Reporting has two very different costs.

CSV exports are searches with scrolling or point-in-time paging over potentially large result sets, followed by streaming the result into a stored report. They are Elasticsearch-heavy and memory-light in Kibana, but a large export is a long-running search that holds resources for its duration.

PDF and PNG reports launch a headless Chromium browser inside the Kibana host, load the dashboard as a user would, wait for every panel to render and capture the page. They are CPU and memory heavy on the Kibana host, and they also generate all the dashboard's searches.

In 8.x reporting jobs are scheduled through Task Manager, so they compete with rules for worker capacity on any instance that runs background tasks. Patterns that keep reporting from hurting everything else:

• Keep scheduled PDF reports off peak hours, and stagger them rather than scheduling everything at 09:00 on Monday.
• Give Chromium room: hosts that render PDFs need memory headroom beyond the Node.js heap, because the browser runs as separate processes.
• Disable reporting job polling on instances that should not render, with xpack.reporting.queue.pollEnabled: false, so only designated instances claim reporting jobs. This is the reporting-specific version of the role split above.
• Make sure every instance shares xpack.reporting.encryptionKey, or reports fail on whichever instance claims them.

Reporting jobs are bounded by a timeout (xpack.reporting.queue.timeout, 4 minutes by default). A dashboard that takes longer to render than the timeout fails as a report even though it eventually loads in a browser. The fix is a lighter dashboard for reporting, not a longer timeout.

Spaces, saved objects and change control

Spaces

A space is a namespace for saved objects and a unit of access control. Roles grant Kibana feature privileges per space, and a space can hide features it does not need. Spaces are the right tool for separating teams on one Kibana, and they cost little: a space is a saved object.

What spaces do not do is isolate data. A data view in a team's space can still query any index the user's Elasticsearch role allows. Data isolation comes from Elasticsearch role index privileges and, where needed, document level security. See /guides/elasticsearch-security-architecture/ and /guides/elasticsearch-multi-tenancy-patterns/.

Saved objects as code

At scale, hand-built dashboards drift, get deleted, and break when a field is renamed. Treat the important ones as code with the saved objects API:

POST /api/saved_objects/_export
{
  "type": ["dashboard"],
  "includeReferencesDeep": true,
  "excludeExportDetails": false
}

This returns NDJSON containing the dashboards and, with includeReferencesDeep, the visualizations, data views and other objects they reference. Commit that file. To deploy it to another Kibana or space:

POST /s/team-payments/api/saved_objects/_import?overwrite=true
Content-Type: multipart/form-data

file=@dashboards.ndjson

The /s/<space-id>/ prefix targets a space; without it you target the default space. overwrite=true replaces objects with the same id. The alternative, createNewCopies=true, regenerates ids, which avoids conflicts but breaks any link that relied on the old ids. Pick one convention and keep it. Use POST /api/spaces/_copy_saved_objects to copy between spaces in the same Kibana.

Every Kibana request that changes state needs a kbn-xsrf header when called from outside the browser. Automation that forgets it gets a 400.

The number of saved objects matters at upgrade time. On startup after an upgrade, Kibana migrates its saved object indices to the new version's format, and the time this takes grows with the number of objects. Tens of thousands of abandoned visualizations make every upgrade slower. Periodically find and delete unused objects:

GET /api/saved_objects/_find?type=visualization&per_page=100&fields=title&sort_field=updated_at

Upgrades and version discipline

Kibana and Elasticsearch upgrades have one Kibana-specific rule: all Kibana instances pointed at a cluster must run the same version, and Elastic's upgrade guidance is to shut down all instances of the old version before starting the new one. A new version migrates saved objects to its format on first start; an old-version instance still running against those indices can fail or write objects the new version then has to deal with.

The sequence that works:

1. Upgrade Elasticsearch first, following /guides/elasticsearch-rolling-upgrades/. Kibana must not be newer than Elasticsearch.
2. Take a snapshot that includes the Kibana feature state, so saved objects can be restored if a migration goes wrong.
3. Stop every Kibana instance.
4. Start one instance of the new version and watch its log until saved object migration completes and /api/status reports available.
5. Start the remaining instances.
6. Check Task Manager health on each, since a backlog of overdue tasks builds while all instances are down.

Expect a short Kibana outage for users during step 3 to 4. Kibana is not upgraded in a rolling fashion across versions.

Failure modes and anti-patterns

Users logged out at random. Symptom: sessions drop when the load balancer moves a user. Cause: xpack.security.encryptionKey differs between instances or is unset. Confirm by checking each instance's config or keystore. Fix by setting the same key everywhere and restarting.

Rules or connectors failing with decryption errors. Cause: xpack.encryptedSavedObjects.encryptionKey differs between instances, or was changed without keeping the old value. Confirm in Kibana logs, which name the failing saved object. Fix by setting one key everywhere. Objects encrypted with a key that is now lost cannot be recovered; their secrets have to be re-entered. When you rotate the key, list the previous ones in xpack.encryptedSavedObjects.keyRotation.decryptionOnlyKeys so existing objects stay readable.

Alerts arriving late. Symptom: rules fire minutes after the event. Confirm with Task Manager runtime.drift p90 and p99, and load near 100%. Fix by adding instances, by reducing expensive rule frequency, or by moving rules off a cold tier.

One rule type dominating. Symptom: high drift with moderate rule counts. Confirm with execution.duration by task type. Fix the rule: narrower index, shorter look-back, lower group cardinality.

PDF reports timing out. Cause: the dashboard takes longer to render than the reporting timeout, often because of a slow panel. Confirm by loading it in a browser and timing the slowest panel. Fix with a reporting-specific lighter dashboard.

Kibana slow, Kibana CPU idle. Cause: Kibana is waiting on Elasticsearch. Confirm with slow logs on the cluster and search thread pool queues. Fix the queries or the cluster, not Kibana.

Cloned instance UUID. Symptom: Stack Monitoring shows fewer Kibana instances than you run, or capacity estimates look wrong. Cause: copied data directories. Fix by giving each instance a fresh data directory.

Auto-refresh on a wall of dashboards. Fifty users with a 20-panel dashboard on a 10 second refresh is 50 x 20 / 10 = 100 searches per second from one dashboard. Confirm with search rates during office hours versus nights. Fix with sensible refresh intervals on shared screens and fewer, more focused panels.

Runbook: Kibana is slow or alerts are late

1. Ask every instance directly for GET /api/status. Any instance not available comes out of the pool.
2. Ask every instance for GET /api/task_manager/_health. Note status, drift p99, load p90 and workload.overdue.
3. If drift and load are high on all instances, background work exceeds capacity. Short term: add an instance with the same config and keys. Medium term: find the expensive task types in execution.duration.
4. If drift is normal but users complain, look at Elasticsearch: GET _cat/thread_pool/search?v&h=node_name,active,queue,rejected and the search slow log.
5. Check that the cluster itself is healthy: GET _cluster/health. A yellow or red cluster, or a node under heap pressure, slows every Kibana request.
6. Check reporting: are PDF jobs running during peak hours on instances that also serve users? Move them with xpack.reporting.queue.pollEnabled: false on the interactive instances.
7. After any change, re-read the Task Manager health on every instance and confirm drift returns to its baseline.

Key takeaways

• Kibana instances are stateless front ends over Elasticsearch-held state; scale out by adding identical instances.
• Set xpack.encryptedSavedObjects.encryptionKey, xpack.security.encryptionKey and xpack.reporting.encryptionKey to the same values on every instance.
• Use /api/status for load balancer health checks and /api/task_manager/_health for capacity, read per instance.
• Drift is the number that tells you background work is falling behind; size from it and from measured execution durations.
• Split ui and background_tasks roles when you can show interference, not before.
• Treat dashboards as code with export and import, and keep saved object counts down for faster upgrades.
• Stop all Kibana instances before starting a new version.

How Sparced helps

Kibana's slowness is often Elasticsearch's slowness, and that part Sparced covers: it reads cluster health, node stats, thread pools and circuit breakers in one pass, so search queueing and rejections, heap pressure and a yellow or red cluster all appear as findings with the affected nodes and indices named. When the cluster is red or yellow it calls _cluster/allocation/explain for each stuck shard and shows the cluster's own explanation, which matters when a Kibana system index is the one that is unassigned. It does not read Kibana's own APIs, so Task Manager drift and rule execution times are still checks you run against Kibana directly.