Elastic Agent and Fleet vs Beats: Architecture and Migration

Beats and Elastic Agent collect the same data, often with the same code underneath, and both ship to Elasticsearch. The difference is how you operate them. With Beats, every shipper type has its own binary, configuration file, upgrade cycle and set of credentials, and you manage them with your configuration management tooling. With Elastic Agent and Fleet, one agent per host runs whatever inputs its policy says, configuration and upgrades are pushed centrally from Kibana, and integrations install their own index templates, ingest pipelines and dashboards. The short answer for most new deployments in 8.x and 9.x is Elastic Agent managed by Fleet; standalone Agent when hosts cannot reach a Fleet Server or configuration must live in Git; and Beats when you have a working estate, no need for Agent-only integrations, and no appetite for a migration. This article covers the architecture, Fleet Server sizing, policy design, and a migration plan that avoids both gaps and duplicates.

The mental model: one agent, many components, one policy

Beats are single-purpose shippers: Filebeat for files and logs, Metricbeat for metrics, Winlogbeat for Windows event logs, Auditbeat, Packetbeat and Heartbeat. Each reads its own YAML file, keeps its own state (Filebeat's registry of file offsets), and has its own modules. Each Beat talks directly to Elasticsearch, Logstash or Kafka with credentials you give it. Nothing coordinates them.

Elastic Agent is a single supervisor process on the host. It receives a policy and runs the components needed to satisfy it: for log files, a component built from Filebeat code; for system metrics, one built from Metricbeat code; for endpoint protection, Elastic Defend; for synthetic monitoring, the synthetics runtime; and so on. When the policy changes, the agent starts, stops or reconfigures components accordingly. You never edit their configs directly.

An integration is a package: a definition of inputs and variables (what to collect, from where), plus the Elasticsearch and Kibana assets for that data (index templates, ingest pipelines, ILM policies, dashboards). Adding an integration to a policy does two things: it installs the assets once in the cluster, and it adds an integration policy (the configured inputs) to the agent policy.

Fleet is the management application in Kibana. Fleet Server is the control-plane service agents talk to: they enroll through it, check in with it, and receive policy changes and actions (upgrade, unenroll, diagnostics) from it. Fleet Server itself runs as an Elastic Agent with the Fleet Server integration, and it stores its state in Elasticsearch system indices.

The data path is separate from the control path:

control:  Kibana (Fleet UI) -> Elasticsearch (.fleet-* indices) <- Fleet Server <-> agents (port 8220)
data:     agents -> Elasticsearch (or Logstash, Kafka, a remote Elasticsearch) as configured by the policy's output

This matters for capacity and for failure: if Fleet Server is down, agents keep collecting and shipping data with the policy they last received. They just cannot receive changes. If Elasticsearch is down, data stops (agents retry and buffer within limits), regardless of Fleet Server.

Data streams and naming

Integrations write to data streams named <type>-<dataset>-<namespace>, for example logs-nginx.access-production or metrics-system.cpu-default. The namespace is a field you set on the integration policy, and it is your main tool for separating data by environment, team or retention. Different namespaces become different data streams, which can carry different ILM policies and different privileges. See index templates and data streams.

Beats in 8.x also write to data streams, but named after the Beat and version (for example filebeat-8.x.y), with one shared template per Beat. This difference is one of the main things to plan for in a migration: the indices, templates, ingest pipelines and dashboards are not the same.

What you gain and what you give up

The decision rule most teams arrive at:

• New deployment, Kibana available, more than a handful of hosts: Elastic Agent with Fleet.
• Air-gapped segments, strict change control, GitOps, or Kubernetes where you already template manifests: standalone Agent. You still get integrations and their data streams, without a control plane.
• A large, stable Beats estate with no need for Agent-only features: Beats remain supported in 8.x and 9.x. Migrate when you need something Agent gives you, not because Beats are old.
• A security use case needing endpoint protection: Agent, because Elastic Defend runs only under Agent.

Fleet Server architecture and sizing

How agents talk to Fleet Server

Each agent holds a long-lived check-in connection to Fleet Server on port 8220 by default. Fleet Server answers when it has something for the agent (a new policy revision, an action) or when the poll times out, and the agent immediately checks in again. Agents also report their status and component health through check-ins.

Fleet Server reads and writes Fleet's state in Elasticsearch. It authenticates to Elasticsearch with a service token for the elastic/fleet-server service account, which grants exactly the privileges Fleet Server needs. Do not run Fleet Server with a superuser credential. See security architecture.

Deployment options

• Elastic Cloud: Fleet Server is provided as part of the Integrations Server. You size it in the deployment.
• Self-managed: run Fleet Server on dedicated hosts or containers, at least two, behind a load balancer that agents reach with a stable URL. The URL is configured in Fleet settings and embedded in enrollment, so choose a DNS name you can keep, not a host name.

Fleet Server needs a TLS certificate whose name matches the URL agents use. Agents verify it. Self-signed certificates work if the agents are given the CA at enrollment.

Sizing

Elastic publishes a Fleet Server scaling table that maps agent counts to recommended CPU and memory for Fleet Server instances, along with the Elasticsearch capacity Fleet's indices need at those scales. Use it; the numbers move between releases as Fleet Server improves, so they are not reproduced here.

The arithmetic around the table is straightforward. Assume:

• 18,000 agents now, growing 30% in a year: plan for 23,400.
• The scaling guide's row for your chosen instance size covers, for the sake of this example, 7,500 agents per instance (read the real number from the guide for your version).

Then:

23,400 / 7,500 = 3.12, round up to 4 instances for load
add 1 for the loss of an instance (N+1): 5 instances

Two other effects are worth planning for:

• Reconnect storms. If Fleet Server restarts, every agent it served reconnects at once. With fewer, larger instances, each restart moves more agents onto the survivors. N+1 is the minimum; spread instances across zones.
• Policy change fan-out. Changing a policy assigned to 18,000 agents means 18,000 agents fetching a new revision. Fleet Server handles this, but it is the moment Fleet's Elasticsearch indices and Fleet Server CPU peak. Make large policy changes outside the busiest hours.

Agent monitoring is data too

Each agent policy has monitoring settings that make agents ship their own logs and metrics. This is very useful for troubleshooting and it is not free. Estimate it:

agents x (agent log volume per day + agent metrics volume per day)

If, for illustration, each agent produced 5 MB per day of its own logs and metrics, 18,000 agents would add 90 GB per day before replicas. Measure your own: enroll ten agents with monitoring on, read the size of the logs-elastic_agent* and metrics-elastic_agent* data streams after a day, and multiply. Many teams keep monitoring on but give those data streams a short retention. See capacity planning for a logging cluster.

Designing agent policies

Each agent has exactly one policy. A policy contains one or more integration policies and the settings that apply to all of them: output, monitoring, and agent-level options. This constraint drives the design.

Patterns that work

• Policies by role, not by host. A "linux-web" policy (system, nginx), a "linux-db" policy (system, PostgreSQL), a "windows-server" policy (system, Windows event logs). Hosts join the policy that matches their role. Tens of policies, not thousands.
• Namespaces for environment. Use the namespace on integration policies (production, staging) to separate data, rather than duplicating policies per environment with otherwise identical content. When environments differ in collection too, separate policies are justified.
• One policy per output destination when outputs differ. The output is a policy-level setting. If some hosts must ship to a separate cluster, or through Logstash, they need their own policy.
• Keep a small "baseline" integration set. Every policy has system metrics and logs configured the same way, so dashboards and alerts work across roles.

Patterns that hurt

• A policy per host. It defeats the point of central management, and every change becomes a script over hundreds of policies.
• One giant policy for everything. Hosts run inputs that find nothing (an nginx input on a database server), log errors, and the policy becomes impossible to change safely.
• Changing a production-wide policy without staging it. A bad integration variable in a policy assigned to 18,000 agents breaks 18,000 agents at once. Copy the policy, assign a canary group, then apply.

Outputs

Policies can send data to Elasticsearch, to Logstash, and in recent 8.x releases to Kafka, or to a remote Elasticsearch cluster. Use Logstash or Kafka when you need an external buffer, routing to non-Elastic destinations, or heavy transformation that you do not want in ingest pipelines. Note that integrations rely on their own ingest pipelines in Elasticsearch; when the agent ships through Logstash, configure Logstash's Elasticsearch output to write to the data stream and let the integration's pipeline run there, rather than re-parsing in Logstash. See ingest pipelines vs Logstash and Logstash pipeline architecture.

Enrollment and upgrades

Enrolling a host into Fleet:

sudo ./elastic-agent install \
  --url=https://fleet.example.internal:8220 \
  --enrollment-token=<token for the target policy>

The enrollment token determines the policy the agent joins. Tokens are created per policy in Fleet and can be revoked; treat them as secrets, because anyone holding one can enroll a host into that policy. After enrollment the agent authenticates with credentials Fleet generated for it, not with the token.

Fleet's own API lets you automate the same steps from your provisioning system:

GET kbn:/api/fleet/agent_policies

GET kbn:/api/fleet/agents?kuery=status:offline

(The kbn: prefix is how Kibana Dev Tools addresses Kibana APIs; from a script, call the Kibana URL directly with an API key.)

Upgrades are pushed from Fleet. Select agents (by policy, tag or query), choose a version, and set a rollout window so upgrades are spread over a period rather than all at once. Version rules:

• Upgrade the stack first. Elasticsearch, then Kibana, then Fleet Server.
• Fleet Server must be on a version equal to or newer than the agents it manages, and agents should not be newer than Kibana.
• Upgrade agents in canary batches: a few hosts per role, watch status and data for a day, then the rest.

See rolling upgrades for the order across the whole stack.

When standalone Agent makes sense

A standalone agent reads its full configuration from elastic-agent.yml on the host and never talks to Fleet Server. You get one binary and integrations' data model, without a control plane. A minimal fragment:

outputs:
  default:
    type: elasticsearch
    hosts: ["https://es01.example.internal:9200"]
    api_key: "<id>:<key>"

inputs:
  - id: system-metrics
    type: system/metrics
    use_output: default
    data_stream:
      namespace: production
    streams:
      - metricsets: [cpu]
        data_stream:
          dataset: system.cpu
        period: 10s

You do not need to write these by hand: in Kibana, adding an integration to a policy and choosing the standalone option shows the generated YAML, which you can then template in your configuration management.

Standalone is right when:

• hosts cannot reach Fleet Server (air-gapped networks, one-way data diodes, strict segmentation),
• configuration must be reviewed and versioned in Git as the source of truth,
• you deploy on Kubernetes and already manage manifests with Helm or Kustomize,
• you want the integration data model but your organization already has a mature fleet management tool.

The costs: you must install integration assets in the cluster separately (through Kibana's Integrations UI or API), distribute and rotate output credentials yourself, and you lose Fleet's central view of agent health. Integration assets and agent configs must be kept in step by you when integrations are upgraded.

Migrating from Beats to Elastic Agent

A migration has four risks: a gap (data not collected while switching), duplicates (data collected twice), broken consumers (dashboards, alerts and saved searches pointing at filebeat-*), and broken parsing (custom Beats processors or Logstash pipelines that the integration does not replicate).

Step 1: inventory

For each Beat configuration, list:

• inputs and modules in use, and the integration that replaces each one (most Beats modules have a matching integration; check that the fields you rely on exist in the integration's mapping),
• custom processors (add_fields, drop_event, dissect, scripts) and where they must move: integration policy processors, a custom ingest pipeline, or Logstash,
• outputs and credentials,
• every dashboard, alert rule, saved search and external consumer that queries filebeat-*, metricbeat-* and similar patterns.

The last list is the one teams forget. Integrations write to logs-* and metrics-* data streams with integration-specific field names in places, so consumers must be repointed.

Step 2: build the policies and verify on canaries

Create agent policies by role, add the integrations, port custom processors, and enroll a few canary hosts alongside their existing Beats. Compare the data side by side for a day: event counts per source, field coverage, parsing failures in the ingest pipelines (look for error.message fields and the integration's failure handling).

Running both on canaries produces duplicates, but only in the canary set, in different data streams, and only for the comparison period.

Step 3: handle file state

Filebeat tracks how far it has read each file in its registry. Elastic Agent does not inherit Filebeat's registry. When an agent starts reading a log path, it begins from its own state, which means already-shipped files may be read again from the beginning. Plan for it explicitly:

• configure the integration's input to ignore files older than a short window, so only recent files are read,
• or accept a bounded duplicate window and tell consumers about it,
• and stop the Beat on a host in the same change that starts the agent's log input, not days apart.

Recent releases have added mechanisms for carrying file state across in specific cases; check the current documentation for your version before depending on them, and test on a canary.

Step 4: cut over by role, in batches

For each role:

1. Enroll the hosts into the agent policy.
2. Stop and disable the Beats on those hosts.
3. Verify event rates in the new data streams match the old rates.
4. Repoint dashboards, alerts and consumers.

Batch by role and by environment, starting with the least critical.

Step 5: decommission

After the old data streams age out under their ILM policies (or you snapshot them for archive), remove the Beats packages, their templates and their credentials. Revoke the Beats API keys or users, so nothing can write to the old indices again.

Worked example: migration overlap cost

Assume 2,000 web hosts, each shipping 1.2 GB per day of nginx and system logs through Filebeat, with 1 replica.

• Normal daily ingest: 2,000 x 1.2 GB = 2,400 GB of primary data, 4,800 GB with the replica.
• Canary overlap: 20 hosts for 2 days in parallel adds 20 x 1.2 x 2 = 48 GB of primary data, 96 GB with the replica. Negligible.
• If instead you ran all 2,000 hosts in parallel for a week "to be safe", you would add 2,400 x 7 = 16,800 GB of primary data, 33,600 GB with replicas, about 34 TB. That is why the cutover runs in batches with a short overlap per batch, not as a cluster-wide parallel run.

Failure modes

Agents show "unhealthy" or "degraded" after a policy change. Cause: an integration input that cannot start (wrong path, missing permissions, unreachable service). Confirm with the agent's status detail in Fleet, which names the failing component, or request a diagnostics bundle. Fix: correct the integration variables, or move the affected hosts to a policy without that integration.

Agents show "offline" in Fleet but data still arrives. Cause: agents cannot reach Fleet Server (network change, certificate expiry, load balancer issue) while the data path to Elasticsearch still works. Confirm by checking Fleet Server reachability from an affected host on port 8220. Fix: restore the control path; agents reconnect on their own. Data was never at risk, but policy changes and upgrades were not received.

No data from new agents, Fleet shows them healthy. Cause: the policy's output points somewhere the agents cannot reach, or the output credential lacks privileges. Confirm in the agent's logs. Fix: correct the output in Fleet settings; agents receive the change on their next check-in.

Dashboards empty after migration. Cause: dashboards still query filebeat-*. Fix: switch to the integration's dashboards or repoint the data views to logs-* patterns.

Duplicate events after cutover. Cause: the Beat was not stopped, or the agent re-read old files. Confirm by checking for the same log.file.path and log.offset values in both the old and new data streams. Fix: stop the Beat, bound the agent's file age, and delete the duplicate range if it matters.

Fleet Server overloaded during a mass upgrade. Cause: every agent upgraded at once. Fix: use a rollout window, and upgrade in batches.

Write rejections in the cluster after onboarding a large group. Cause: a new wave of agents, plus the integration's ingest pipelines, adds both write load and ingest CPU. Confirm with write thread pool rejections and ingest node CPU. Fix: onboard in batches, and size ingest capacity for the integration pipelines. See thread pools and rejections.

Runbook: rolling out Fleet to an existing estate

1. Stack ready. Elasticsearch and Kibana on the target version; Fleet set up; a stable Fleet Server URL in DNS; TLS certificate issued for it.
2. Fleet Server deployed. At least two instances behind a load balancer, sized from Elastic's scaling guide for your planned agent count with N+1. Fleet Server uses a service token, not a superuser.
3. Policies designed. By role, with namespaces for environment, baseline system integration in each, output set per policy.
4. Agent monitoring retention set. Short ILM retention on logs-elastic_agent* and metrics-elastic_agent*, sized from a 10-agent measurement.
5. Canaries enrolled alongside Beats. Compare counts and fields for a day.
6. Consumers listed and repointed. Dashboards, alerts, saved searches, external readers.
7. Cutover by role in batches. Enroll, stop Beats in the same change, verify rates.
8. Upgrades on a rollout window. Fleet Server first, then agents in canary batches.
9. Decommission. Remove Beats, their templates and their credentials after the old data ages out.

Key takeaways

• Agent is one supervisor running components chosen by a single policy; Fleet Server is the control path, Elasticsearch is the data path, and losing Fleet Server does not stop data.
• Integrations bring their own templates, ingest pipelines and dashboards, and write to <type>-<dataset>-<namespace> data streams; Beats data lives elsewhere, so consumers must move.
• Size Fleet Server from Elastic's scaling guide, then add growth and N+1, and expect reconnect storms on restarts.
• Design policies by role with namespaces for environment; never one per host.
• Standalone Agent fits air-gapped, GitOps and Kubernetes-manifest estates. Beats remain a supported choice for stable estates.
• Migrate with canaries side by side, stop the Beat in the same change that starts the agent input, and plan for file re-reads because Agent does not inherit Filebeat's registry.

How Sparced helps

Sparced looks at the Elasticsearch side of an Agent or Beats estate, not at the agents themselves. In one read-only pass it reports what onboarding thousands of agents tends to do to a cluster: write thread pool rejections per node, nodes past 85% heap, disk approaching the watermarks, ILM policies with errors (for example on new integration data streams), and, if the cluster is yellow or red, the allocation explanation for each stuck shard with a fix command and a check. It does not read Fleet, agent status or policies, and it installs nothing on hosts or the cluster.