Elasticsearch Node Roles and Cluster Topology for Production

A production Elasticsearch cluster fails in predictable ways: the elected master is starved by data work, a two-zone layout loses its quorum with one zone, a coordinating node runs out of heap reducing a giant aggregation, or a zone outage piles every replica onto the survivors and fills their disks. Each of these is a topology decision made early and paid for later. This article explains how the cluster coordination layer actually works, what each node role does, how to size masters, and how to lay out nodes across zones so that the loss of a node or a zone is an event you planned for.

The mental model: one elected master, a voting configuration, and everyone else

Every Elasticsearch cluster has exactly one elected master at a time. It owns the cluster state: the list of nodes, every index's mappings and settings, and the routing table that says which shard copy lives on which node. Only the elected master changes cluster state. It publishes each new version to every node, and a change is committed once a quorum of the voting configuration has accepted it.

The voting configuration is the set of master-eligible nodes whose votes count. Since 7.0, Elasticsearch manages it automatically: when master-eligible nodes join or leave, the elected master adjusts the configuration. A quorum is a strict majority of it:

quorum = floor(voting_nodes / 2) + 1

Two consequences drive every design below:

1. An even number of master-eligible nodes buys nothing. With four, the cluster still tolerates only one failure. Elasticsearch deliberately leaves one node out of the voting configuration when the count is even, so that a network split cannot leave two equal halves.
2. Cluster state publication is on the critical path. Every mapping update, index creation, shard allocation and rollover is a cluster state change that waits for a quorum to acknowledge it. Slow, overloaded or distant master-eligible nodes make every one of those operations slow.

When there is no elected master, nothing that changes cluster state can happen. Searches against shards whose location is already known may continue for a while, but the cluster health API returns a master_not_discovered_exception, index creation fails, and the cluster cannot recover from any further shard failure.

Bootstrapping, once

A brand-new cluster needs to know which nodes form its first voting configuration. That is cluster.initial_master_nodes, and it applies only the first time a cluster forms:

# elasticsearch.yml on each master-eligible node, first start only
cluster.name: prod-logs
node.name: master-a
node.roles: [ master ]
discovery.seed_hosts: [ "master-a.internal", "master-b.internal", "master-c.internal" ]
cluster.initial_master_nodes: [ "master-a", "master-b", "master-c" ]

Remove cluster.initial_master_nodes from the configuration once the cluster has formed. Leaving it in place is a known way to bootstrap a second, separate cluster by accident if nodes are later restarted with fresh data directories. discovery.seed_hosts stays: it is how restarting and new nodes find the cluster.

Inspect the current voting configuration directly:

GET _cluster/state?filter_path=metadata.cluster_coordination

{
  "metadata": {
    "cluster_coordination": {
      "term": 14,
      "last_committed_config": ["Xk2...", "pQ9...", "b7L..."],
      "last_accepted_config": ["Xk2...", "pQ9...", "b7L..."],
      "voting_config_exclusions": []
    }
  }
}

The values are node IDs. term increases with every election; a term that keeps climbing on a healthy-looking cluster means masters are being re-elected repeatedly, which is a stability problem to investigate in the master logs.

Node roles, one by one

Roles are set with node.roles in elasticsearch.yml. If you do not set it, a node gets every role except voting_only: master, data, data_content, data_hot, data_warm, data_cold, data_frozen, ingest, ml, remote_cluster_client and transform. That default is fine for a laptop and wrong for most production clusters.

Read the roles of a running cluster:

GET _cat/nodes?v&h=name,node.role,master,heap.percent,heap.max,ram.max,cpu,disk.used_percent&s=node.role

The node.role column is a string of single-letter abbreviations: m master, v voting_only, d data, s data_content, h data_hot, w data_warm, c data_cold, f data_frozen, i ingest, l ml, r remote_cluster_client, t transform, and - for a coordinating-only node. The master column shows * on the elected master.

master

Master-eligible nodes take part in elections and in cluster state publication. A dedicated master has only this role:

node.roles: [ master ]

Dedicated masters exist to isolate cluster coordination from data work. A data node that is also master-eligible can be elected master, and then a long garbage collection pause caused by a heavy aggregation, or a saturated disk during a merge, stalls the whole cluster's state changes. A common rule of thumb is to move to three dedicated masters once a cluster grows beyond a handful of data nodes, or earlier if it holds many indices or changes mappings frequently. Elastic's documentation recommends dedicated master-eligible nodes for large or high-churn clusters; it does not set a single node-count threshold.

Master nodes need persistent storage (they store cluster state on disk), but not much of it.

voting_only

A voting-only node is master-eligible but can never be elected. It votes in elections and acknowledges cluster state publications. It must also carry the master role:

node.roles: [ master, voting_only ]

It can additionally hold data roles, but its main use is as a tiebreaker: the third vote that lets two full masters in two zones survive the loss of either one. Because it is on the publication path, it needs low-latency networking and reliable storage, but it can run on modest hardware since it never does the master's work.

Data roles and data tiers

data is the generic role: the node can hold any shard on any tier. The tier-specific roles are data_content, data_hot, data_warm, data_cold and data_frozen. Use either the generic role or tier roles on a given node, not both, to keep allocation predictable.

• data_content: non-time-series indices (catalogs, entities). Regular indices default to this tier through index.routing.allocation.include._tier_preference: data_content.
• data_hot: the write path for time-series data. Data stream backing indices default to data_hot.
• data_warm, data_cold: older, less-queried data, usually on denser storage.
• data_frozen: partially mounted searchable snapshots only. Run frozen nodes with data_frozen and nothing else.

A typical hot node carries both data_hot and data_content, so that regular indices have somewhere to live:

node.roles: [ data_hot, data_content, ingest ]
node.attr.zone: eu-west-1a

The tier mechanics, ILM movement and the frozen cache are covered in /guides/elasticsearch-data-tiers-ilm/.

Coordinating-only

A node with an empty role list is coordinating-only:

node.roles: [ ]

Every node coordinates the requests it receives: it fans a search out to the relevant shards, then runs the reduce phase, merging the shard results, sorting hits and combining aggregation buckets. Large terms aggregations or wide date histograms make the reduce phase memory-intensive. A coordinating-only tier takes that load off data nodes and gives Kibana and applications a stable endpoint.

Do not add them by reflex. Every node, coordinating or not, is part of the cluster and must receive and acknowledge cluster state updates, so Elastic's documentation warns that adding too many coordinating-only nodes increases the burden on the whole cluster. Two or three behind a load balancer is a common pattern; dozens is not. Many clusters do fine sending client traffic straight to data nodes.

ingest

Ingest nodes run ingest pipelines before documents are indexed. Any node with the ingest role can run them, and with the default role set every node has it. Dedicated ingest nodes help when pipelines are CPU-heavy (large grok patterns, enrichment, script processors) and you want that cost off the nodes that index and search. Size them for CPU, not disk. The trade-offs are in /guides/ingest-pipelines-vs-logstash/.

ml

Machine learning jobs run in native processes outside the JVM, on nodes with the ml role. Elastic's documentation recommends dedicated ML nodes that also carry remote_cluster_client so datafeeds can read from remote clusters:

node.roles: [ ml, remote_cluster_client ]

ML memory is bounded by xpack.ml.max_machine_memory_percent, which defaults to 30% of the machine's memory, so the JVM heap and native ML memory must both fit in RAM. On dedicated ML nodes this setting is often raised; leave enough for the JVM and the operating system.

transform

Transform nodes run continuous and batch transforms. As with ML, the documented pattern is [ transform, remote_cluster_client ] on dedicated nodes when transform load is significant. Light transform usage can share nodes with other roles.

remote_cluster_client

Required on a node that sends cross-cluster search or cross-cluster replication requests to remote clusters. If you use CCS or CCR, make sure the nodes receiving those requests have it. See /guides/elasticsearch-cross-cluster-replication-search/.

Sizing the masters

Master work scales with cluster state size and churn, not with data volume. The cluster state grows with the number of indices, shards and mapped fields. Churn grows with index creation, rollover, mapping updates from dynamic fields, and shard movements.

What a dedicated master needs:

• Heap. The full cluster state lives on heap. Elastic's shard sizing documentation gives a general rule of thumb: reserve at least 1 GB of heap on each master-eligible node for every 3,000 indices in the cluster. Treat it as a planning figure and measure your own cluster state size before trusting it.
• CPU. Modest. Two to four cores are typical for dedicated masters in clusters of moderate size (rule of thumb).
• Disk. Small but reliable and reasonably fast; cluster state is persisted on every commit.
• Network. Low latency to every other node, especially other master-eligible ones.

Worked example

Assumptions: 4,000 indices (data stream backing indices plus regular indices), about 9,000 shards, 40 data nodes.

Step 1, heap from the rule above: 4,000 / 3,000 = about 1.3 GB of master heap for index metadata. Add headroom for the JVM's own needs and for publication buffers. A 4 GB heap (on an 8 GB machine, keeping heap at no more than 50% of RAM) leaves ample margin.

Step 2, check the actual size. The cluster state is visible through GET _cluster/state, but on a large cluster that response itself is big; fetch it once, off-peak, and measure the payload rather than viewing it in Kibana.

Step 3, sanity check the trend. If index count is growing by hundreds a week because of daily indices for many sources, the master's problem is the shard design, not its heap. See /guides/elasticsearch-shard-sizing/.

Heap rules that apply to every node: set -Xms and -Xmx equal, keep heap at no more than 50% of RAM, and below the compressed ordinary object pointers threshold, which is near 32 GB and varies by JVM: 26 GB is safe on most systems, and up to about 30 GB works where the startup log reports compressed ordinary object pointers [true]. In 8.x the default heap is sized automatically from node roles and available memory; on dedicated nodes the automatic sizing is usually reasonable, but confirm it with heap.max in _cat/nodes. More in /guides/elasticsearch-jvm-heap-circuit-breakers/.

Zones: awareness and forced awareness

A cluster spread across availability zones must do two things: keep a primary and its replica in different zones, and keep quorum when a zone disappears.

Shard allocation awareness

Tag each node with its zone using a custom attribute, then tell the allocator to consider that attribute:

# on each node
node.attr.zone: zone-a

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": "zone"
  }
}

With awareness on, the allocator spreads copies of each shard across the values of zone it knows about. With one replica and two zones, the primary is in one zone and the replica in the other.

The zone-failure trap and forced awareness

Plain awareness has a sharp edge. When zone-a fails, every shard that had a copy there now has one copy, in zone-b. The allocator wants to restore the replica count, and the only nodes left are in zone-b, so it allocates the missing replicas there too. zone-b now holds two copies of everything, which needs roughly double the disk it was using. If it does not have that headroom, nodes cross the high disk watermark, then flood stage, and indices start going read-only. You have turned a zone outage into a cluster-wide write outage.

Forced awareness prevents this. You declare the full set of zone values, and the allocator will not place more copies in the surviving zones than they would hold if all zones were present:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": "zone",
    "cluster.routing.allocation.awareness.force.zone.values": "zone-a,zone-b,zone-c"
  }
}

With forced awareness, losing a zone leaves the affected replicas unassigned and the cluster yellow until the zone returns. That is the intended trade: degraded redundancy, but no disk exhaustion. Size each zone to carry its share of the data plus normal headroom; you do not need to size it for double.

Quorum across zones

Shard copies are only half of the problem. The masters must keep a quorum.

• Three zones, one master-eligible node in each. Losing any zone leaves 2 of 3. This is the clean design.
• Two zones only. Any split of master-eligible nodes across two zones leaves one zone with a majority. Lose that zone and the cluster has no master. Two zones cannot survive the loss of either zone without a third vote somewhere else.
• Two zones plus a tiebreaker. Put one full master-eligible node in each main zone and a voting-only node in a third location. Adding more full masters to the two main zones does not help, because one zone always ends up holding the majority. Losing either main zone leaves the other master plus the tiebreaker, 2 of 3. This is the pattern Elastic uses for two-zone deployments on its hosted service, and it works on self-managed clusters if you have a third location with low enough latency.

# third location, tiebreaker
node.name: tiebreaker-c
node.roles: [ master, voting_only ]
node.attr.zone: zone-c

Do not stretch one cluster across regions. Cluster coordination and replication assume low-latency links; across regions, use cross-cluster replication and cross-cluster search instead.

Reference topologies

Changing master nodes safely

Replacing or removing master-eligible nodes is where people lose quorum by hand. The rule: never remove half or more of the voting configuration at once.

To remove master-eligible nodes deliberately, first exclude them from the voting configuration, wait for the exclusion to take effect, then stop them:

POST _cluster/voting_config_exclusions?node_names=master-old-1,master-old-2

The call returns once the cluster has reconfigured to exclude those nodes. Confirm with GET _cluster/state?filter_path=metadata.cluster_coordination. Then stop the excluded nodes. When the old nodes are gone for good, clear the exclusions list so it does not affect future nodes with the same names:

DELETE _cluster/voting_config_exclusions

The number of exclusions is capped by cluster.max_voting_config_exclusions, which defaults to 10.

For a rolling replacement of three masters, the safe order is: add the new master-eligible node, wait for it to join, then exclude and remove one old node, and repeat. You never drop below three voting nodes. You do not need voting exclusions for a normal restart of a master-eligible node in a rolling upgrade, as long as you restart one at a time and wait for it to rejoin.

When quorum is already lost

If a majority of master-eligible nodes is permanently gone (disks destroyed, not just stopped), the remaining nodes cannot elect a master. Bring the lost nodes back with their data directories if at all possible. The elasticsearch-node command-line tool has an unsafe-bootstrap mode that forces a new cluster from a surviving node, and a detach-cluster mode to move other nodes onto it. Both are, as their names say, unsafe: they can lose recent cluster state changes and data. Restoring from snapshot onto a fresh cluster is often the better choice. Treat this as a last resort and follow /guides/elasticsearch-red-cluster-runbook/.

Failure modes and anti-patterns

Master-eligible data nodes in a busy cluster. Symptom: index creation and rollover take tens of seconds; logs on the master mention slow cluster state publication or nodes failing to acknowledge in time; occasional master re-elections. Confirm: _cat/nodes shows the elected master (*) also has d or tier roles and high heap or CPU; term in cluster coordination metadata keeps increasing. Fix: add three dedicated masters, then remove the master role from data nodes one at a time, using voting exclusions if you remove more than one quickly.

Two master-eligible nodes, or two zones without a tiebreaker. Symptom: losing one node or one zone leaves the cluster without a master. Confirm: count voting nodes in last_committed_config; map them to zones. Fix: add a third master-eligible node, or a voting-only tiebreaker in a third location.

Awareness without forced awareness on a full cluster. Symptom: after a zone outage, the surviving zone hits disk watermarks and indices get the read_only_allow_delete block. Confirm: awareness attributes set, no force values; _cat/allocation shows survivor disks climbing. Fix: set forced awareness values for all zones. See /guides/elasticsearch-disk-watermarks/.

Leaving cluster.initial_master_nodes in place. Symptom: usually nothing, until a master-eligible node is rebuilt with an empty data directory and forms a new one-node cluster with the same name. Confirm: check elasticsearch.yml on master-eligible nodes. Fix: remove the setting after first formation.

Coordinating-only node out of memory on aggregations. Symptom: circuit breaker exceptions or heap pressure on coordinating nodes during dashboard loads. Confirm: GET _nodes/stats/breaker on those nodes; look at the request and parent breakers. Fix: give them enough heap for the reduce phase; reduce bucket counts; use composite aggregations for exhaustive bucket iteration.

Default roles on every node. Symptom: ML jobs start on data nodes and compete for memory; every node is master-eligible, so an even count or a large voting configuration slows publication. Confirm: _cat/nodes shows long role strings everywhere. Fix: set explicit node.roles on every node.

Checklist: designing or reviewing topology

1. Count master-eligible nodes in the voting configuration. It must be odd and at least 3 for production.
2. Map each voting node to a zone. No single zone may hold a majority unless you accept losing the cluster with that zone.
3. If only two zones are available, place a voting-only tiebreaker in a third location.
4. Set node.roles explicitly on every node. No production node runs with the default role set.
5. Use dedicated masters once the cluster has more than a few data nodes, many indices, or frequent mapping and rollover activity.
6. Size master heap from cluster state, not data volume; keep heap at no more than 50% of RAM and below the compressed-oops threshold (26 GB is safe on most systems).
7. Set node.attr.zone on every node, enable awareness, and set forced awareness values for every zone.
8. Size each zone's disk for its share plus headroom below the 85% low watermark.
9. Add coordinating-only nodes only for a measured reduce-phase problem, and keep their number small.
10. Remove cluster.initial_master_nodes after the cluster first forms.
11. Rehearse a master replacement with voting exclusions in a staging cluster before you need it.

Key takeaways

• One elected master commits every cluster state change through a quorum of the voting configuration; slow masters make everything slow.
• Odd numbers of master-eligible nodes, at least three, one per zone. Two zones need a voting-only tiebreaker in a third.
• Set node.roles explicitly: dedicated masters, tier roles on data nodes, [ ] for coordinating-only, dedicated ml and transform when they are busy.
• Awareness spreads copies across zones; forced awareness stops a zone outage from filling the survivors' disks.
• Remove masters with voting configuration exclusions, one safe step at a time, and remove cluster.initial_master_nodes after bootstrap.

How Sparced helps

Sparced reads the nodes, cluster health, shards and allocation endpoints in one pass, so it shows each node's roles, which node is the elected master, heap and disk per node, and how shard copies are spread across nodes. When a node is past 85% heap or the cluster is red or yellow, it makes the follow-up calls, including _cluster/allocation/explain for each stuck shard, and prints the cluster's own explanation, such as an awareness decider refusing a placement. Each finding lists the actual nodes and shards, a fix command with those names filled in, and a check to confirm the change. It installs no agent and writes nothing, so topology changes stay a decision you make.