Elasticsearch Cross-Cluster Replication and Search Architecture

Once a deployment outgrows one cluster, two separate questions appear, and they are often confused. The first is how to query data that lives in several clusters, which is cross-cluster search (CCS). The second is how to keep a copy of an index in another cluster, for disaster recovery or to put data close to its readers, which is cross-cluster replication (CCR). Both depend on the same remote cluster connection, and both fail in ways that are easy to predict once you know how that connection works. This article covers the connection modes and their security models, how CCS executes a search and what it costs in latency, how CCR followers pull changes from leaders and what makes them fall behind, and a failover procedure that works because it was written before the outage.

The remote cluster connection

A remote cluster is registered on the local cluster under an alias. The local cluster opens connections to it and keeps them alive; CCS and CCR then refer to the remote by that alias. The connection is always initiated by the cluster doing the reading: the cluster that runs a cross-cluster search, or the follower in CCR. The leader never connects to the follower.

There are two connection modes.

Sniff mode (the default). You give a list of seed nodes. The local cluster connects to a seed, retrieves the remote cluster state, and opens connections directly to a subset of the remote's eligible nodes, by default up to 3 of them, set by cluster.remote.<alias>.node_connections. Every local node that needs the connection must be able to reach those remote nodes by the addresses they publish.

PUT _cluster/settings
{
  "persistent": {
    "cluster.remote.dc-west.mode": "sniff",
    "cluster.remote.dc-west.seeds": ["10.20.0.11:9300", "10.20.0.12:9300"]
  }
}

Proxy mode. You give a single address, typically a load balancer or a TCP proxy, and the local cluster opens a fixed number of socket connections to it, by default 18, set by cluster.remote.<alias>.proxy_socket_connections. The remote nodes do not need to be individually reachable. This is the mode to use when the remote is behind NAT, in another cloud account, or exposed through a single endpoint, as Elastic Cloud deployments are.

PUT _cluster/settings
{
  "persistent": {
    "cluster.remote.dc-west.mode": "proxy",
    "cluster.remote.dc-west.proxy_address": "es-west.internal.example:9300",
    "cluster.remote.dc-west.server_name": "es-west.internal.example"
  }
}

server_name is sent as the TLS SNI value, which lets a proxy route to the right cluster.

Two more settings change behavior during failures:

• cluster.remote.<alias>.skip_unavailable: when true, a cross-cluster search skips a remote that cannot be reached and returns results from the others, marking it as skipped. When false, the whole search fails. The default changed to true in recent 8.x releases; earlier releases defaulted to false. Set it explicitly so the behavior does not depend on version.
• cluster.remote.<alias>.transport.ping_schedule and cluster.remote.<alias>.transport.compress: keepalive pings for connections through firewalls that drop idle flows, and compression for links where bandwidth costs more than CPU.

Check the connection:

GET _remote/info

{
  "dc-west": {
    "connected": true,
    "mode": "proxy",
    "proxy_address": "es-west.internal.example:9300",
    "server_name": "es-west.internal.example",
    "num_proxy_sockets_connected": 18,
    "max_proxy_socket_connections": 18,
    "initial_connect_timeout": "30s",
    "skip_unavailable": true
  }
}

connected: false, or a connected count below the maximum, is the first thing to check whenever CCS or CCR misbehaves.

Security models

There are two ways to authorize traffic between clusters.

Certificate-based (the original model). Clusters trust each other's transport TLS certificates, and the remote connection uses the transport port (9300 by default). The user running a cross-cluster search must exist, with suitable roles, on both clusters: the local cluster authenticates the user and the remote cluster authorizes the request against roles of the same name defined there. This couples the security configuration of both clusters tightly, and any user with transport-level trust gets broad reach.

API key based (the newer model, generally available in recent 8.x releases). The remote cluster exposes a separate remote cluster server port, 9443 by default, enabled with remote_cluster_server.enabled: true. The remote cluster's administrator creates a cross-cluster API key that states exactly which indices may be searched and which may be replicated:

POST _security/cross_cluster/api_key
{
  "name": "from-dc-east",
  "access": {
    "search": [ { "names": ["logs-*", "metrics-*"] } ],
    "replication": [ { "names": ["orders-*"] } ]
  }
}

The encoded key is stored in the local cluster's keystore as cluster.remote.<alias>.credentials, and the remote is configured to use port 9443. Authorization is then the intersection of what the local user may do and what the key allows, and the remote administrator controls the ceiling without having to mirror local roles. For new deployments on current versions, this model is easier to reason about and audit. The surrounding security design is in security architecture.

Cross-cluster search: how it executes

A cross-cluster search names indices with the remote alias as a prefix. Wildcards work on both parts:

GET logs-*,dc-west:logs-*,dc-*:logs-*/_search
{
  "size": 20,
  "query": { "match": { "message": "timeout" } },
  "aggs": { "by_service": { "terms": { "field": "service.name" } } }
}

The coordinating node on the local cluster sends work to each remote. Two execution strategies exist, chosen by ccs_minimize_roundtrips, which defaults to true:

• Minimize roundtrips (default). The local coordinator sends one request per remote cluster. Each remote runs the whole search on its own shards, reduces its results, and returns a single response. The local coordinator merges these. Cross-cluster traffic is one request and one response per remote, which is the right choice whenever there is meaningful latency between clusters.
• Without minimizing roundtrips. The local coordinator treats remote shards as if they were local and communicates with them through each phase (query, then fetch, plus any can-match prefilter). That means several round trips across the link, each paying its latency, but more precise global reduction of things like terms aggregations.

Latency arithmetic for the default mode, with stated assumptions: a 70 ms round trip to a remote region, the search taking 120 ms on the remote's own shards and 90 ms locally.

1. Local and remote work run in parallel.
2. Remote path: 70 ms network + 120 ms search = 190 ms, plus transfer of the response.
3. Local path: 90 ms.
4. Total is roughly the slowest path plus the final merge: about 200 ms.

Without minimizing roundtrips, the same search pays at least two round trips to the remote (query and fetch), about 140 ms of pure network, plus the remote shard work, so it is noticeably slower over a WAN. Keep the default unless you have a measured reason.

The response reports per-cluster outcome in _clusters:

"_clusters": {
  "total": 3,
  "successful": 2,
  "skipped": 1,
  "running": 0,
  "partial": 0,
  "failed": 0,
  "details": {
    "(local)":  { "status": "successful", "took": 91 },
    "dc-west":  { "status": "successful", "took": 188 },
    "dc-south": { "status": "skipped", "failure": [ { "reason": { "type": "connect_transport_exception" } } ] }
  }
}

Applications that rely on CCS must read _clusters.skipped. With skip_unavailable: true, a search that silently omits a whole region returns a normal-looking response with fewer hits. A dashboard showing a sudden traffic drop from a region may simply be a skipped cluster.

Kibana data views can include remote patterns (dc-*:logs-*), and ES|QL also supports remote index patterns in recent versions. Version compatibility for CCS is limited: a local cluster can search remotes on the same major version and, with restrictions, the last minor of the previous major. Check compatibility before upgrading either side; the order matters. See rolling upgrades.

CCS design patterns

• Search hub. A small cluster with no data of its own, connected to every regional data cluster, serving Kibana. It keeps users off the data clusters' coordinating nodes and centralizes saved objects. It needs coordinating capacity sized for the merge phase, not storage.
• Regional data, global view. Data stays in the region where it was produced, for residency or bandwidth reasons, and only queries cross regions.
• Archive cluster. Older data on a cheaper cluster, searched via CCS alongside current data. A frozen tier on the same cluster is usually simpler; see data tiers with ILM.

Cross-cluster replication: leader, follower and the pull model

CCR replicates an index from a leader cluster to a follower cluster. It is index-level and active-passive per index: the follower index is read-only and changes only by replicating from its leader. CCR is a feature of the paid subscription levels (Platinum and Enterprise on self-managed, and the equivalent on Elastic Cloud); check your subscription before designing around it.

How it works underneath:

1. Bootstrap. When you create a follower, it first copies the leader's segment files using a remote recovery, similar to a snapshot restore from the leader. That transfer is throttled by ccr.indices.recovery.max_bytes_per_sec, 40mb by default, per follower node.
2. Replay. Each follower shard then repeatedly reads batches of operations from the corresponding leader shard, using the leader's history of operations retained through soft deletes, and applies them in order. The follower pulls; the leader serves.
3. Retention leases. The follower holds a retention lease on each leader shard, which tells the leader not to discard operation history the follower still needs. If the follower is unreachable for longer than the lease period, index.soft_deletes.retention_lease.period, 12h by default, the leader may discard that history. The follower then cannot resume and must be re-bootstrapped.

Soft deletes are enabled by default for indices created since 7.0, and CCR requires them on the leader.

Create a follower manually:

PUT orders-2026/_ccr/follow?wait_for_active_shards=1
{
  "remote_cluster": "dc-east",
  "leader_index": "orders-2026"
}

The follower's mappings and many settings follow the leader. You cannot write to the follower, and you should not change its replicated settings.

Auto-follow patterns

For time-based indices and data streams, new leader indices are created continuously, and creating followers by hand does not scale. An auto-follow pattern makes the follower cluster check the leader for new indices matching patterns and create followers automatically:

PUT _ccr/auto_follow/east-logs
{
  "remote_cluster": "dc-east",
  "leader_index_patterns": ["logs-*", ".ds-logs-*"],
  "leader_index_exclusion_patterns": ["logs-debug-*"],
  "follow_index_pattern": "{{leader_index}}"
}

Auto-follow only applies to indices created after the pattern exists. Existing indices must be followed manually. Data streams replicate as data streams when their backing indices are auto-followed, and the follower data stream is marked as replicated.

What CCR does not replicate: index templates, component templates, ILM policies, ingest pipelines, users, roles and cluster settings. A follower cluster that will become a primary on failover needs all of these in place beforehand. Manage them as code applied to both clusters, or restore them from a snapshot's global state.

Watching lag

GET _ccr/stats

GET orders-2026/_ccr/stats

Per-shard stats include leader_global_checkpoint, follower_global_checkpoint, operations_read, operations_written, time_since_last_read_millis, read_exceptions, outstanding_read_requests, and fatal_exception.

• Operation lag is leader_global_checkpoint - follower_global_checkpoint. It is a count of operations, not time. Translate it to time with the leader's indexing rate: 120,000 operations behind at 20,000 operations per second is 6 seconds behind.
• time_since_last_read_millis that keeps growing on an index with active writes means the follower is not pulling.
• fatal_exception means the shard stopped following and will not recover by itself.
• read_exceptions shows repeated failures to read from the leader, often a connection problem.

GET _ccr/stats also reports auto-follow stats, including recent errors for patterns that failed to create followers, a common silent failure when the follower cluster cannot create an index (for example because of the shard limit).

Lag grows when the follower cannot keep up: the follower's write path is slower than the leader's, the link is saturated, or follow parameters limit batch size and concurrency. The follow request accepts tuning parameters such as max_read_request_operation_count, max_outstanding_read_requests and max_write_buffer_size; the defaults are reasonable, and the more common fix is giving the follower comparable hardware to the leader. A follower is a full indexing workload, not a passive copy.

ILM with CCR

ILM can run on both sides. On the leader, rollover and the rest of the policy run normally. On the follower, follower indices are not rolled over by ILM; ILM's unfollow action, which ILM applies automatically to a follower index before rollover, shrink and searchable snapshot actions, waits until the leader has marked the index as complete and then converts the follower into a normal index. Keep policies consistent on both sides: a follower whose ILM deletes an index sooner than the leader, or never, is a common surprise in DR drills.

Failover and failback

CCR does not fail over automatically. Failover is a procedure you run, and it is only as good as the last time you rehearsed it.

Failover to the follower

Assume the leader cluster dc-east is down and dc-west holds followers.

1. Declare. Decide that dc-east is lost for long enough to fail over. Make sure nothing can still write to it, so you do not end up with two divergent primaries.
2. Stop following. For each follower index: pause, close, unfollow, open. Unfollowing converts it into a regular, writable index.

POST orders-2026/_ccr/pause_follow
POST orders-2026/_close
POST orders-2026/_ccr/unfollow
POST orders-2026/_open

3. Data streams. Promote the replicated data stream so it accepts writes, then roll it over so the write index is a normal index rather than a former follower:

POST _data_stream/_promote/logs-app-default
POST logs-app-default/_rollover

4. Delete or pause auto-follow patterns pointing at the lost leader, so they do not try to recreate followers when it returns.
5. Point writers and readers at dc-west. DNS, load balancer or client configuration. Writers using aliases or data stream names need no index name change.
6. Validate. Index a test document, search it, check templates and ILM policies exist and match.

Data loss on failover is the replication lag at the moment of failure. With healthy CCR, that is typically seconds of operations, and you can bound it from your lag monitoring.

Failback

The old leader cannot simply resume. Its indices may contain operations the new primary never received, and the new primary has received writes the old leader lacks. The usual procedure:

1. Bring dc-east back, but keep writers away from it.
2. Delete or rename its stale copies of the failed-over indices.
3. Create followers on dc-east that follow the new leaders in dc-west, bootstrapping a full copy.
4. When lag is near zero, schedule a planned switchover: stop writes, wait for lag to reach zero, unfollow on dc-east, redirect writers, then re-establish dc-west as follower.

For time-based data, a shortcut is often acceptable: let dc-east start fresh indices, and let older data age out on whichever side holds it, reading across both with CCS meanwhile.

Active-active

CCR is active-passive per index, but two clusters can each lead some indices and follow others. The standard pattern is region-specific index names: dc-east writes orders-east-* and follows orders-west-*; dc-west does the reverse. An alias or a search across both patterns gives a global view in each region. No document is ever written in two places, so no conflict resolution is needed. What this pattern does not give you is a single writable index shared by both regions; Elasticsearch has no multi-master replication.

Failure modes and anti-patterns

Follower stuck after a long outage. Symptom: fatal_exception mentioning that requested operations are missing on the leader. The follower was disconnected longer than the retention lease period and the leader discarded the history. Fix: unfollow and delete the follower, then follow again to re-bootstrap. Prevent it by monitoring time_since_last_read_millis.

Remote connection silently down. Symptom: CCS results missing a region, CCR lag growing. Confirm: GET _remote/info shows connected: false. Typical causes: firewall idle timeouts (set a ping schedule), certificate expiry, or sniff mode across a NAT where remote publish addresses are unreachable (use proxy mode).

Auto-follow not creating followers. Symptom: new leader indices have no follower. Confirm: auto-follow stats in GET _ccr/stats show recent errors. Typical causes: follower cluster at cluster.max_shards_per_node, missing privileges, or the pattern created after the index. Fix the cause; follow existing indices by hand.

DR cluster without templates or policies. Symptom: after failover, the first rollover produces an index with default mappings, or ILM never runs. Fix: manage templates, pipelines, ILM policies and security as code on both clusters.

Undersized follower. Symptom: lag grows during peak and recovers overnight. The follower cannot index as fast as the leader. Fix: follower data nodes comparable to the leader's hot tier.

Assuming CCR is a backup. A deletion on the leader is replicated. Keep snapshots on both sides; see snapshot and restore.

Runbook: checks before you need them

Monthly:

• GET _remote/info from every cluster that reads from another: all connected: true, socket or node counts at their maximum.
• GET _ccr/stats: no fatal_exception, no growing read_exceptions, operation lag translated into seconds is within your RPO, and no auto-follow errors.
• Compare templates, component templates, ILM policies and ingest pipelines between leader and follower clusters.
• Confirm skip_unavailable is set explicitly on every remote, to the value your applications expect.
• Run a CCS query from Kibana against every remote and check _clusters.skipped is 0.

Quarterly:

• Run a failover drill on one non-critical index or data stream: pause, close, unfollow, open, promote, write, then re-follow it.
• Time the drill and update the documented RTO.
• Check subscription level and remote cluster version compatibility before any planned upgrade.

Key takeaways

• One remote cluster connection serves both CCS and CCR. Sniff mode connects to remote nodes directly; proxy mode connects through one address and is the choice behind NAT or a load balancer.
• Prefer API key based remote access on current versions: the remote decides exactly what can be searched or replicated.
• CCS with the default ccs_minimize_roundtrips pays one round trip per remote. Always read _clusters.skipped, and set skip_unavailable explicitly.
• CCR followers pull operations from leader shards and hold retention leases; an outage longer than the lease (12h by default) forces a re-bootstrap.
• Lag is an operation count; convert it to seconds with the indexing rate to get your RPO.
• CCR does not replicate templates, policies, pipelines or security, and it does not fail over by itself. Rehearse the procedure.

How Sparced helps

Sparced reads cluster health, nodes, shards and allocation in a single pass, so a follower cluster with unassigned or unhealthy shards appears as findings with the index and shard names attached rather than a bare yellow. When the cluster is red or yellow, it runs _cluster/allocation/explain for each stuck shard and prints the cluster's own explanation, which covers problems such as follower data nodes past a disk watermark. It also reads security configuration as part of the pass. Each finding carries a fix command with names filled in and a check to confirm it; Sparced does not configure remote clusters, start or stop following, or write anything to the cluster.