Elasticsearch Rolling Upgrades: 7.17 to 8.x and 8.x to 9.x

An Elasticsearch upgrade fails in one of three ways: something the new version refuses to start with (a removed setting, an index too old to read), a cluster that sits yellow for hours because shards cannot move between versions, or an application that breaks because its client spoke an API the new version dropped. All three are predictable. The answer is a fixed sequence: clear every critical deprecation before you touch a node, upgrade the nodes in the order Elastic documents (data tiers from coldest to hottest, master-eligible nodes last), control allocation so each restart is a short recovery instead of a full rebalance, and accept up front that the only rollback is a snapshot taken before you started. This article walks the sequence, explains why each step exists, and covers the 7.17 to 8.x and 8.x to 9.x paths specifically.

The mental model: what a mixed-version cluster can and cannot do

A rolling upgrade means the cluster runs two versions at once for a while. Three rules govern that period, and almost every upgrade problem is one of them.

A newer node can join a cluster with an older master, but not the reverse. The elected master publishes cluster state, and a node must understand that state to join. A newer node understands older state. An older node cannot be guaranteed to understand state produced by a newer master. That is why master-eligible nodes go last: if the master is upgraded early, any older node that restarts may be unable to rejoin.

Shards can move from older nodes to newer nodes, not from newer to older. A newer node writes Lucene segments in a format the older node may not read. Once a primary lives on an upgraded node, its replica cannot be allocated to a node still running the old version; in allocation explain output this is the node_version decider saying no. During a rolling upgrade this is expected as long as you keep moving forward; it becomes a problem only if you stall halfway with too few upgraded nodes to hold the replicas.

Data written by the new version is not readable by the old version. Elastic does not support downgrading a node. Once an upgraded node has opened its data directory, that directory belongs to the new version, which is why rollback is a snapshot.

A fourth rule governs which majors you can cross: Elasticsearch reads indices created in the current or the previous major only. 8.x reads indices created in 7.x and 8.x, and refuses to start if it finds an index created in 6.x. 9.x reads indices created in 8.x; indices created in 7.x or earlier must be reindexed, deleted, or marked read-only (a write block, which the Upgrade Assistant can apply) before you upgrade. Snapshots follow the same lineage. An index that has been carried forward through rolling upgrades still carries its original creation version, so a cluster that started life on 6.x and was rolled to 7.17 can still hold 6.x-created indices that 8.x will not open.

The supported paths

You cannot skip a major, and you cannot cross a major from an arbitrary minor.

Two practical points. First, always take the latest patch of the bridge minor (7.17.latest, 8.19.latest). Bridge releases carry the deprecation checks and migration tooling for the next major, and later patches fix bugs in exactly that tooling. Second, minor upgrades within a major use the same procedure with far fewer surprises; the deprecation API still matters because deprecations accumulate and become removals at the next major.

Before the window: deprecations, system indices, old indices

The deprecation info API

Elasticsearch tells you what will break. The deprecation info API reports every setting, index and configuration that the next major will reject.

GET _migration/deprecations

A trimmed response looks like this:

{
  "cluster_settings": [],
  "node_settings": [
    {
      "level": "critical",
      "message": "Setting [node.master] is deprecated",
      "url": "...",
      "details": "Remove the [node.master] setting. Set [node.roles] and include the [master] role.",
      "resolve_during_rolling_upgrade": false
    }
  ],
  "index_settings": {
    "logs-2019.06.01": [
      {
        "level": "critical",
        "message": "Index created before 7.0",
        "details": "This index was created with version 6.8.23 and is not compatible with 8.0. Reindex or remove the index before upgrading.",
        "resolve_during_rolling_upgrade": false
      }
    ]
  },
  "ml_settings": []
}

The fields that matter:

• level: critical means the next major will not start, or will not work correctly, until this is resolved. warning means behavior changes or a future removal. Treat every critical as a blocker. Read every warning and decide.
• resolve_during_rolling_upgrade: when false, you must fix this before the first node is upgraded. When true, it can be fixed while the upgrade is in progress, typically because the fix is per node.
• index_settings: keyed by index name. Old-version indices appear here. These are the slowest items to fix because each one is a reindex.

You can scope the call to an index pattern to keep the output readable on large clusters:

GET logs-*/_migration/deprecations

The API reads configuration, not traffic. It cannot see that your application still sends a request that the next major removed. For that, use the deprecation log. Elasticsearch logs a deprecation warning each time a deprecated feature is used, and when cluster.deprecation_indexing.enabled is true these entries are also indexed into the .logs-deprecation.elasticsearch-default data stream, which Kibana's Upgrade Assistant reads. Each entry carries the request's X-Opaque-Id header if the client set one, which is how you find which application is calling the deprecated endpoint. If your clients do not set X-Opaque-Id, ask the teams to add it before the upgrade; it turns "something is still using typed APIs" into "the billing indexer is."

Responses to deprecated calls also carry a Warning HTTP header; logging it in staging is a cheap client audit.

System indices and the Upgrade Assistant

Elasticsearch and Kibana keep internal state in system indices (security, tasks, Kibana saved objects, ML, Watcher and others). Their mappings and format must also be on a version the next major accepts. The feature migration API reports and performs this:

GET _migration/system_features

{
  "features": [
    { "feature_name": "security", "minimum_index_version": "7.17.0", "migration_status": "NO_MIGRATION_NEEDED" },
    { "feature_name": "kibana", "minimum_index_version": "6.8.0", "migration_status": "MIGRATION_NEEDED" }
  ],
  "migration_status": "MIGRATION_NEEDED"
}

If the top-level migration_status is MIGRATION_NEEDED, start it and poll the GET until it reports NO_MIGRATION_NEEDED:

POST _migration/system_features

On the bridge minor, the Kibana Upgrade Assistant wraps all of this: it lists critical deprecations, runs the system feature migration, and offers to reindex old indices (or, for 8.x to 9.x, to mark 7.x-created indices read-only). It is the documented route and worth using even if you script everything else.

Reindexing old indices: the arithmetic

For an index created in the previous-previous major, you have three choices: reindex it into a new index, delete it if retention allows, or (for the 8.x to 9.x hop) mark it read-only. For time-series data, deleting is often the right answer: if you keep 30 days of logs and the oldest 6.x index is 400 days old, it is a retention bug, not a migration task.

When you must reindex, estimate time before you plan the window. Assume:

• 1.2 TB of primary data in old indices
• a sustained reindex throughput of 40 MB/s across the cluster (measure yours with a single test index; it depends on mapping, ingest pipelines and hardware)

Then: 1.2 TB is 1.2 x 1,024 x 1,024 = about 1,258,000 MB. Divided by 40 MB/s is about 31,450 seconds, or 8.7 hours, before replicas build. Replicas copy after the primary is done, at recovery speed, not reindex speed. Plan reindexing as a separate change days before the upgrade, not as part of the upgrade window.

The reindex itself, with slicing and the alias swap that keeps clients unchanged, is covered in zero-downtime reindexing.

Running the upgrade, node by node

Prepare the cluster

Before the first node goes down:

1. Take a snapshot of everything, including system indices via feature states. This is the rollback. Confirm it finished with state: SUCCESS, not PARTIAL. See snapshots and disaster recovery.
2. Get the cluster green. Do not start a rolling upgrade on a yellow cluster; you will not be able to tell upgrade-induced yellow from pre-existing yellow, and a missing replica plus a restarting node can become a missing primary.
3. Check disk headroom. Shards that move during the upgrade need space on the target. If nodes are near the high watermark (90% by default), relocations stall. See disk watermarks.
4. Confirm the version spread you start from:

GET _cat/nodes?v=true&h=name,version,node.role,master&s=version

Every node should be on the same version before you begin.

5. Check plugins. Plugins must match the Elasticsearch version exactly. Every node needs the new plugin build installed during its upgrade, or it will not start.

6. Upgrade the test cluster first, with a restored copy of production indices.

Disable replica allocation

Each data node goes through the same loop. The goal is to make the node's absence a short pause rather than a rebalance. Elastic documents this step as optional but recommended.

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}

cluster.routing.allocation.enable accepts all (the default), primaries, new_primaries and none. Setting primaries means that when the node stops, Elasticsearch promotes in-sync replicas to primary where needed but does not start building new replicas elsewhere. Without this, the cluster sees a node leave and starts copying gigabytes to restore redundancy, then copies them back when the node returns.

Delayed allocation (index.unassigned.node_left.delayed_timeout, default 1 minute) already delays replica rebuild briefly, but a package upgrade and JVM start often take longer than a minute on a large node. Setting primaries explicitly removes the race.

Optionally flush

POST _flush

A flush commits Lucene segments and trims the translog, so the node has less to replay on start. Elastic notes recovery is much faster if you also pause non-essential indexing. Synced flush is gone: _flush/synced was deprecated in 7.6 and removed in 8.0. Scripts from the 6.x era that call it will fail against 8.x.

Pause machine learning, if you run it

POST _ml/set_upgrade_mode?enabled=true

This halts ML jobs and datafeeds and blocks new jobs from opening, so they are not reassigned repeatedly as nodes restart. Turn it off at the end with enabled=false; forgetting to is a classic post-upgrade ticket.

Stop, upgrade, start

On a package install, this is the familiar sequence:

sudo systemctl stop elasticsearch
# install the new package version for your platform
# install the matching version of every plugin
sudo systemctl start elasticsearch

Before starting, diff elasticsearch.yml and jvm.options against the new version's defaults. For a 7.x to 8.x hop in particular, the node will refuse to start if it finds removed settings. The common ones:

• legacy role settings (node.master, node.data, node.ingest and similar) were removed in 8.0; use node.roles
• discovery.zen.* settings from the pre-7.0 discovery system
• settings flagged critical in the deprecation output for node settings, which you will already have a list of

Custom jvm.options lines deserve the same scrutiny: Elasticsearch bundles its JDK, and an old hand-tuned GC flag the new JVM does not recognize stops the node at boot. Prefer a file in jvm.options.d/ containing only the heap size.

Confirm the node rejoined

GET _cat/nodes?v=true&h=name,version,node.role,master

The upgraded node should appear with the new version.

Re-enable allocation and wait

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": null
  }
}

Setting it to null returns it to the default all. Then wait for green before the next node:

GET _cluster/health?wait_for_status=green&timeout=120s

GET _cat/recovery?v=true&active_only=true&h=index,shard,stage,source_node,target_node,bytes_percent,translog_ops_percent

Do not move to the next node while recoveries are active. Overlapping two nodes' worth of missing replicas is how a rolling upgrade turns into a red cluster.

Why recovery is usually fast

When a node rejoins, its replicas are behind by whatever was indexed while it was away. Since 7.x, Elasticsearch uses peer recovery retention leases: the primary retains the operation history (soft deletes) needed to bring a returning copy up to date, so recovery can replay only the missed operations rather than copying files. Leases expire after index.soft_deletes.retention_lease.period, 12 hours by default. If a node is down longer than that, its copies fall back to file-based recovery, which copies whole segments. Keep each node's absence short, and do not leave a node half-upgraded overnight.

Recovery speed is bounded by indices.recovery.max_bytes_per_sec, 40 MB/s by default on most nodes (Elasticsearch derives a different default for dedicated cold and frozen nodes), and by cluster.routing.allocation.node_concurrent_recoveries, 2 by default. Raising them shortens the upgrade on fast hardware, but recovery competes with live traffic. Watch latency and set them back afterwards.

Upgrade order: coldest data first, masters last

Elastic documents the order explicitly. Upgrade data nodes tier by tier, starting with frozen, then cold, then warm, then hot, then other data nodes (such as data_content), then the remaining nodes that are neither master-eligible nor data nodes (dedicated ingest, ML, transform, coordinating-only), and master-eligible nodes last, including voting-only nodes.

The reasoning follows directly from the mixed-version rules:

• Cold before hot: ILM moves indices from hot towards cold. If hot nodes were upgraded first, shards on new-version hot nodes could not move to old-version warm or cold nodes, which the node_version decider forbids, and ILM stalls. Upgrading in the direction of data flow means data only ever moves onto same-version or newer nodes.
• Masters last: an older node cannot join a cluster whose elected master is newer. If you upgrade masters first, every remaining old node becomes unable to rejoin after its own restart.

Elastic's guide does not order the master-eligible nodes among themselves; a common practice is to upgrade the non-elected ones first and the elected master last, so the cluster holds only one extra election. Check which one is elected:

GET _cat/master?v=true

When you stop the elected master, a new election happens among the remaining eligible nodes. With three master-eligible nodes, stopping one leaves two of three votes, so quorum holds. Never stop two master-eligible nodes at once in a three-node master set. The voting rules are covered in cluster topology.

Worked example: how long will it take

Assume a cluster of 3 dedicated masters, 12 hot nodes, 6 warm nodes, 2 coordinating nodes. Per data node: 6 minutes to stop, install, and start; recovery after rejoin averages 9 minutes (mostly operation-based, some file-based on busy shards). Per non-data node: 6 minutes, no recovery.

• Data nodes: 18 nodes times (6 + 9) minutes = 270 minutes
• Coordinating: 2 times 6 = 12 minutes
• Masters: 3 times 6 = 18 minutes, plus a short election pause for the last one
• Total: about 300 minutes, five hours, of serial work

That is the honest number to put in the change request. Two levers shorten it. The first is not part of Elastic's documented procedure, which is one node at a time, so treat it as a rule of thumb you prove on a test cluster: with shard allocation awareness across zones, so no zone holds two copies of a shard, some teams upgrade all nodes of one tier in one zone together. With 3 zones and the tier order kept, warm becomes 3 batches of 2 and hot 3 batches of 4: 6 batches instead of 18 restarts, each with a longer recovery. The second lever is the recovery throttle, discussed above.

Clients, Kibana and the rest of the stack

Upgrade order across the stack

Upgrade Elasticsearch first, then Kibana, then the ingest and collection layer (Fleet Server and Elastic Agent, Logstash, Beats). Kibana will not run against an Elasticsearch older than itself, which is why it follows.

Kibana is not a rolling upgrade. When Kibana starts on a new version, it migrates its saved objects, and running two Kibana versions against the same Elasticsearch at once is unsupported. Stop every Kibana instance, upgrade them, then start them. Plan a short Kibana outage even when Elasticsearch stays up; it grows with the number of saved objects.

For Fleet-managed agents, Fleet Server must be on a version equal to or newer than the agents it manages, and agents should not be newer than Kibana. Upgrade Fleet Server before the agents. See Elastic Agent and Fleet versus Beats.

Language clients and REST API compatibility

The client question has two parts: will the old client work against the new server, and will your code work with the new client.

For 7.x clients against 8.x, Elasticsearch 8.x supports REST API compatibility: a client that sends the header Accept: application/vnd.elasticsearch+json; compatible-with=7 (and the matching Content-Type) gets 7.x request and response behavior for the APIs that changed. The 7.17 language clients can be configured to send this header. That lets you upgrade the cluster first and the applications later, so the risky change is not coupled to every application deploy.

The same mechanism carries forward to 9.x with compatible-with=8; compatibility is honored across one major only. It is a bridge, not a destination: it covers request and response formats for changed endpoints, does not resurrect removed features, and does not cover the transport client, which was removed in 8.0.

For Java specifically, the High Level REST Client was deprecated in 7.15 in favor of the Java API Client. Migrating application code from one to the other is real work and is best done as its own project, using the HLRC compatibility mode as the bridge while you upgrade the cluster.

Security defaults on 8.x

8.0 enables security by default for new installations, and its behavior for upgraded clusters depends on what your 7.x configuration said explicitly. If your 7.x cluster ran without security by default rather than by an explicit setting, read the 8.0 upgrade notes on security and set xpack.security.enabled explicitly in elasticsearch.yml to whatever you intend. The design of the security layer itself is in security architecture.

Choosing a strategy

A rolling upgrade is the default, but not the only option.

The rollback reality

Write this sentence in the change request: there is no in-place downgrade.

• Downgrading is not supported. An upgraded node's data directory cannot be opened by the old version.
• Indices created after the upgrade are in a format the old version cannot read.
• A snapshot taken on the newer version cannot be restored to an older version. Snapshots move forward only.

So the only rollback is: build or keep a cluster on the old version, and restore the snapshot you took before starting. Everything indexed after that snapshot is lost unless you can replay it from a source (Kafka, Logstash persistent queues with replay capability, or the application). This shapes the plan:

• Take the pre-upgrade snapshot as late as possible, ideally minutes before the first node stops.
• Know whether your ingest can be replayed from a buffer for the length of the upgrade window. A Kafka buffer makes this realistic for logging.
• Accept the point of no return. Elastic's guidance is that once a node runs the target version, the upgrade runs through to the last node, and a mixed-version cluster is supported only for the duration of the upgrade. In a stalled mixed-version cluster, the move is forward: finish upgrading the remaining nodes.

Failure modes

Node will not start after the package upgrade. Symptom: the service fails immediately; the log names an unknown or removed setting, or an index created in an unsupported version. Confirm by reading the last lines of the Elasticsearch log. Fix: remove the setting (it should have been in the node_settings deprecations), or, for an old index, you are on the wrong side of a reindex you skipped. Do not assume the old version can reopen that data directory; restore from snapshot if it cannot.

Cluster stays yellow after a node rejoins. Symptom: replicas unassigned, upgrade paused. Confirm with allocation explain:

GET _cluster/allocation/explain
{
  "index": "logs-app-2026.10.20",
  "shard": 0,
  "primary": false
}

If the decider is node_version ("cannot allocate replica shard to a node with version ... since this is older than the primary version"), the primary is on an upgraded node and there is no other upgraded node to hold the replica. Fix: keep going. The next upgraded node will receive it. If the decider is enable, you forgot to set allocation back to null. If it is disk_threshold, you are out of space on the eligible nodes.

ILM stalls during the upgrade. Symptom: indices waiting in a migrate or allocate step. Cause: the order was hot first, so data cannot move to old-version cold nodes. Fix: continue upgrading the colder tiers. See ILM troubleshooting.

An old node cannot rejoin. Symptom: a node started on the old version reports it cannot join because the master is on a newer version. Cause: masters were upgraded before data nodes, or a data node restarted for an unrelated reason after the master upgrade. Fix: upgrade that node. There is no way to get it to join the newer master on the old version.

Applications break after the upgrade. Symptom: 400 errors on requests that worked yesterday. Cause: removed APIs (typed endpoints, include_type_name, removed query parameters), or a client library too old. Confirm: the deprecation log in staging would have shown these, keyed by X-Opaque-Id. Fix: enable REST API compatibility headers in the client as a stopgap, then fix the code.

Upgrade takes far longer than planned. Cause: file-based recoveries after long node downtime, or the recovery throttle. Confirm with _cat/recovery. Fix: shorter downtime per node, pre-staged packages, a temporary throttle increase.

Runbook: rolling upgrade

Before the window:

1. Confirm the path: source is on the bridge minor's latest patch. Target is a version the bridge supports.
2. GET _migration/deprecations returns no critical items. Every warning has been read.
3. GET _migration/system_features reports NO_MIGRATION_NEEDED.
4. Deprecation log in staging is clean of calls from production applications, or each caller has a plan.
5. Test cluster upgraded with production-shaped data, and the test showed you per-node timings.
6. New packages and matching plugin builds staged on every host.
7. Config diffed against the new version; removed settings deleted.

At the start:

8. Cluster is green. GET _cat/nodes shows one version.
9. Take a snapshot including feature states. Confirm SUCCESS.
10. POST _ml/set_upgrade_mode?enabled=true if ML is in use.

For each node, in order (frozen, cold, warm, hot, other non-master roles, non-elected masters, elected master):

11. PUT _cluster/settings with cluster.routing.allocation.enable: primaries.
12. Optional POST _flush.
13. Stop, upgrade package and plugins, start.
14. GET _cat/nodes shows the node on the new version.
15. PUT _cluster/settings with cluster.routing.allocation.enable: null.
16. GET _cluster/health?wait_for_status=green and _cat/recovery?active_only=true empty, unless yellow is explained by the node_version decider.

After the last node:

17. Cluster green, all nodes on the new version.
18. POST _ml/set_upgrade_mode?enabled=false.
19. Stop all Kibana instances, upgrade, start. Confirm saved object migration finished in the Kibana log.
20. Upgrade Fleet Server, then agents, Logstash and Beats.
21. Reset any temporary recovery throttle changes.
22. Run GET _migration/deprecations again: this time you are reading the list for the next upgrade.

Key takeaways

• Read GET _migration/deprecations and clear every critical item before touching a node; resolve_during_rolling_upgrade: false means before, not during.
• Cross majors only from the bridge minor (7.17 for 8.x; 8.18 for 9.0, 8.19 for 9.1 and later), and only with indices created in the previous major or later.
• Upgrade frozen, cold, warm, hot, then other roles, then masters, elected master last, because shards and cluster state only flow from older to newer.
• Set cluster.routing.allocation.enable to primaries per node and back to null after, and never start the next node while recoveries are active.
• Kibana is stop-all, upgrade, start-all, and it follows Elasticsearch.
• REST API compatibility headers let you upgrade the cluster before the applications.
• Rollback is a snapshot restore to an old-version cluster. Data written after the snapshot must be replayable from somewhere, or it is gone.

How Sparced helps

Sparced is useful at both ends of the upgrade. Before you start, one read-only pass shows whether the cluster is genuinely green, which nodes are past 85% heap, which nodes are near their disk watermarks and whether thread pools are already rejecting, so you do not start a rolling upgrade on a cluster that is already strained. During the upgrade, if the cluster goes yellow or red, Sparced calls _cluster/allocation/explain for each stuck shard and prints the cluster's own reason, so a node_version refusal (keep going) is distinguished from a forgotten allocation setting or a full disk (act now), with the fix command and a check to confirm it. It does not call the deprecation info API or plan the upgrade for you, and it writes nothing to the cluster.