Elasticsearch Index Templates, Component Templates, Data Streams

Most mapping incidents in a time-series cluster are not caused by a bad mapping. They are caused by the wrong template winning, a template change that never reached the index everyone is querying, or a data stream used for data that needed updates. The fix is a clear model: an index template is a recipe applied exactly once, at index creation; component templates are the reusable ingredients; priority decides which recipe applies; and a data stream is a named, append-only sequence of hidden indices created from that recipe at every rollover. This article walks through that model, the APIs that let you see what a template will actually produce before it produces it, and the cases where you should not use a data stream at all.

The mental model: templates act once, at creation

An index template does nothing to an index that already exists. When Elasticsearch is about to create an index, whether because you called PUT my-index, because a bulk request named an index that does not exist yet, or because a data stream rolled over, it looks for templates whose index_patterns match the new name. It picks one, builds the final settings, mappings and aliases from it, and creates the index. After that moment the template and the index are unrelated. Editing the template later changes the next index, never the current one.

That single fact explains most "my template change did nothing" tickets. If you add a field mapping to the template behind logs-app-default, the current write index of that data stream does not get it. The next backing index, created at the next rollover, does.

There are two families of templates in the APIs:

Family	API	Status
Composable index templates	`_index_template`	The current mechanism since 7.8; use it
Component templates	`_component_template`	Building blocks referenced by composable templates
Legacy templates	`_template`	Deprecated; still read, but composable templates take precedence when both match

If a cluster has been upgraded from 6.x or early 7.x, it may still carry legacy templates. When a composable template and a legacy template both match a new index name, the composable one is used and the legacy one is ignored. A cluster that "used to work" and changed behavior after someone created a broad composable template is a common symptom of this.

How a composable template is assembled

A composable template has its own template block (settings, mappings, aliases) and an ordered composed_of list of component templates. The final index configuration is merged in a fixed order, where each later source overrides the earlier ones:

Component templates, in the order listed in composed_of.
The index template's own template block.
Any settings or mappings in the create index request itself.

Mappings merge recursively: two component templates can each contribute different fields under properties, and the result contains both. Where the same setting or the same field is defined twice, the later source wins.

A typical layout separates concerns so each component template has one owner:

PUT _component_template/acme-settings-default
{
  "template": {
    "settings": {
      "index.number_of_shards": 1,
      "index.number_of_replicas": 1,
      "index.lifecycle.name": "acme-logs-30d"
    }
  },
  "_meta": { "owner": "platform-team" }
}

PUT _component_template/acme-mappings-ecs-base
{
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "message":    { "type": "match_only_text" },
        "host":       { "properties": { "name": { "type": "keyword" } } },
        "service":    { "properties": { "name": { "type": "keyword" } } }
      }
    }
  }
}

PUT _component_template/acme-mappings-checkout
{
  "template": {
    "mappings": {
      "properties": {
        "order_id": { "type": "keyword" },
        "amount":   { "type": "scaled_float", "scaling_factor": 100 }
      }
    }
  }
}

PUT _index_template/acme-logs-checkout
{
  "index_patterns": ["logs-checkout-*"],
  "data_stream": {},
  "composed_of": [
    "acme-settings-default",
    "acme-mappings-ecs-base",
    "acme-mappings-checkout"
  ],
  "priority": 200,
  "version": 3,
  "_meta": { "description": "Checkout service logs" }
}

Three habits pay off here. Put version on every template and bump it on every change, because GET _index_template then tells you which revision is live, and you can line that up with the creation dates of backing indices. (An index does not record the template version that created it.) Put an owner in _meta. And keep the index template itself thin, so that the per-service difference is visible in one small component template rather than buried in a 400-line body.

Component templates cannot be deleted while a composable template references them; the delete request fails. That is a safety net, not a nuisance: it stops someone removing the base mappings that 40 index templates depend on.

Priority and pattern overlap

When several composable templates match a new index name, Elasticsearch uses the one with the highest priority. A template without a priority is treated as priority 0. Only one composable template is ever applied; they do not stack. Stacking is what component templates are for.

Elasticsearch refuses to create a composable template whose patterns overlap an existing template with the same priority. The error names both templates. This is deliberate: two equal-priority templates matching the same name would make the winner ambiguous.

Priority becomes important with the built-in templates. Elasticsearch ships built-in index templates for patterns such as logs-*-*, metrics-*-* and synthetics-*-*, each with priority 100, and Fleet integrations install their own templates for specific datasets with overlapping patterns and a priority of up to 200. Elastic's documented advice: if you use Fleet or Elastic Agent, give your own overlapping templates a priority below 100; if you do not, and you deliberately want to take over a pattern such as logs-*, give your template a priority above 500. Three practical rules follow:

If you create a template for logs-* at priority 200, you override the built-in logs-*-* template for every log data stream, including ones created by Elastic Agent integrations, and you lose their mappings and lifecycle. That is rarely what anyone intended.
Match narrowly. logs-checkout-* at priority 200 is safe as long as no installed integration owns the checkout dataset; logs-* at any priority above 100 is a cluster-wide change.
To customize managed data streams, prefer the @custom component templates that the managed templates already reference (for example a <type>-<dataset>@custom or logs@custom template). Those hooks exist so you can add mappings or settings without forking the managed template. Check composed_of on the managed template to see exactly which @custom names it looks for in your version.

See the winner before it wins

Two simulate APIs answer "what will this produce?" without creating anything. The first takes an index name and shows the configuration that would be applied, plus any lower-priority templates that matched and lost:

POST _index_template/_simulate_index/logs-checkout-default

{
  "template": {
    "settings": { "index": { "number_of_shards": "1", "lifecycle": { "name": "acme-logs-30d" } } },
    "mappings": { "properties": { "order_id": { "type": "keyword" } } },
    "aliases": {}
  },
  "overlapping": [
    { "name": "logs", "index_patterns": ["logs-*-*"] }
  ]
}

The overlapping array is the most valuable field in the response. It lists the templates that also matched and were outranked. If you expected the built-in template to apply and it appears here, your own template is shadowing it.

The second simulate API takes a template name, or a full template body that does not exist yet, and shows its resolved output. Use it in CI before applying a change:

POST _index_template/_simulate/acme-logs-checkout

A rule worth enforcing in review: no template change is merged without the _simulate_index output for one representative index name pasted into the change description.

Data streams: what they are underneath

A data stream is a name that clients write to and search against, backed by a sequence of hidden indices. Exactly one backing index is the write index at any time; searches against the data stream name fan out to all backing indices.

Backing indices are named .ds-<data-stream>-<yyyy.MM.dd>-<generation>, for example .ds-logs-checkout-default-2026.10.21-000014. The date is when that backing index was created, and the generation is a six-digit counter that increments at each rollover. Elastic's documentation warns not to rely on the naming pattern for anything functional, so treat it as a readable label only and use GET _data_stream to discover the actual indices.

A data stream needs three things:

A matching composable template that contains a data_stream object (it can be empty, {}).
A @timestamp field mapped as date or date_nanos. If the template does not map it, Elasticsearch maps it as date for you. Every document must carry it.
Append-only writes. Index requests must use op_type=create, and in _bulk you use the create action, not index.

The data stream is created automatically on the first write to a matching name, or explicitly:

PUT _data_stream/logs-checkout-default

GET _data_stream/logs-checkout-default

{
  "data_streams": [
    {
      "name": "logs-checkout-default",
      "timestamp_field": { "name": "@timestamp" },
      "indices": [
        { "index_name": ".ds-logs-checkout-default-2026.10.07-000013", "index_uuid": "..." },
        { "index_name": ".ds-logs-checkout-default-2026.10.21-000014", "index_uuid": "..." }
      ],
      "generation": 14,
      "status": "GREEN",
      "template": "acme-logs-checkout",
      "ilm_policy": "acme-logs-30d",
      "hidden": false,
      "system": false,
      "replicated": false
    }
  ]
}

The fields that matter: indices lists backing indices oldest first, so the last entry is the write index. template tells you which index template the data stream is bound to, which is the template the next rollover will use. status is the worst health of its backing indices. ilm_policy is the policy configured in the template. replicated is true for a data stream that is a cross-cluster replication follower, which cannot be written to until it is promoted.

Updates and deletes

Data streams are append-only by design, but you can still change existing documents:

_update_by_query and _delete_by_query against the data stream name work across all backing indices.
To update or delete a single document by id, address the backing index that holds it, and use if_seq_no and if_primary_term from a prior search hit to avoid racing with other writers.

If your workload does this routinely, rather than as an exception, a data stream is the wrong container. See the decision table below.

Rollover: how a data stream grows

Rollover creates a new backing index, makes it the write index, and leaves the previous one in place for searches. It can be triggered by ILM, by data stream lifecycle, or manually:

POST logs-checkout-default/_rollover

{
  "acknowledged": true,
  "shards_acknowledged": true,
  "old_index": ".ds-logs-checkout-default-2026.10.21-000014",
  "new_index": ".ds-logs-checkout-default-2026.10.21-000015",
  "rolled_over": true,
  "dry_run": false,
  "conditions": {}
}

Manual rollover is the standard way to make a template change take effect now rather than at the next scheduled rollover. Recent 8.x releases also support a lazy rollover (POST <data-stream>/_rollover?lazy=true), which marks the data stream so that the rollover happens on the next write instead of immediately. That avoids creating an empty backing index on a quiet data stream.

The sizing arithmetic of rollover is where shard counts come from. Assume a data stream ingests 300 GB of primary data per day, the template sets index.number_of_shards: 3, and ILM rolls over at max_primary_shard_size: 50gb:

Each backing index holds at most 3 x 50 GB = 150 GB of primary data.
300 GB per day / 150 GB per backing index = 2 rollovers per day.
With a 30-day retention, the data stream holds about 60 backing indices, or 60 x 3 = 180 primary shards, 360 shards with one replica.

If the same stream used max_age: 1d alone, you would get one 300 GB index per day split across 3 shards, 100 GB per shard, which is past the commonly cited 10 to 50 GB range for shard size. Size-based conditions keep shard size predictable when ingest volume changes; age-based conditions keep retention granular. Most policies use both, whichever comes first. The capacity model behind these numbers is in shard sizing.

Changing mappings and settings on a live data stream

You have three tools, from least to most disruptive:

Change	How	Effect
Add a new field, change a dynamic setting	Update the template, then `PUT <data-stream>/_mapping` or `PUT <data-stream>/_settings`	Applies to existing backing indices where the change is compatible
Anything else (new analyzer, shard count, field type)	Update the template, then roll over	Applies from the new backing index onward
Change a field type on historical data	Reindex into a new data stream	Rewrites the data

PUT <data-stream>/_mapping with "write_index_only": true applies the mapping to the write index only, which is useful when an older backing index would reject it.

The trap is field type drift. If response_time was keyword in generation 13 and long in generation 14, a range query across the whole data stream behaves differently per backing index, and aggregations may fail or return partial results for that field. The usual fixes are a runtime field that coerces the old type at query time, or accepting that the old generations age out under ILM.

Data stream lifecycle and ILM

Backing indices can be managed by ILM or, in recent 8.x releases, by the simpler data stream lifecycle, configured in the template:

PUT _index_template/acme-metrics-app
{
  "index_patterns": ["metrics-app-*"],
  "data_stream": {},
  "template": {
    "lifecycle": { "data_retention": "14d" }
  },
  "priority": 200
}

Data stream lifecycle handles rollover and retention automatically and has no phases or tiers. ILM is the choice when you need hot, warm, cold or frozen tiers, force merge at a specific phase, or searchable snapshots. If both are configured, the index.lifecycle.prefer_ilm setting decides which manages a backing index; it defaults to true. Run GET _data_stream/<name> and look at the per-index management fields in your version to confirm which one is in charge, rather than assuming. ILM itself, and how to unstick it, is covered in ILM troubleshooting and data tiers with ILM.

Data stream or plain index: the decision

Data streams are designed for append-only, timestamped data with a retention policy. Several workloads do not fit:

Workload	Container	Why
Logs, metrics, traces, audit events	Data stream	Append-only, `@timestamp`, retention by age
Time-series events that are later corrected in bulk, rarely	Data stream	`_update_by_query` covers the exception
Time-series data updated by id routinely (order status, session state)	Plain indices behind a write alias, with rollover	Updates by id need a known index; data streams make every update a lookup
Product catalog, user profiles, documents for search	Single plain index behind an alias	No time dimension, updated in place, reindexed on mapping change
Metrics with dimensions and high volume	Time series data stream (`index.mode: time_series`)	Storage savings and dimension-based routing; see storage cost optimization
Small reference data (a few thousand lookups)	Single plain index, one shard	Rollover adds nothing

For the "plain indices with rollover" pattern, the template does not have a data_stream object, the first index is created explicitly as my-orders-000001 with a write alias, and ILM rolls it over using index.lifecycle.rollover_alias. It gives you rollover and retention while keeping normal index semantics for updates. For catalogs and search indices, use an alias so that a mapping change can be done by reindexing into a new index and swapping the alias; the full procedure is in zero-downtime reindexing.

You can convert an existing alias-with-rollover setup into a data stream with POST _data_stream/_migrate/<alias>, provided a matching template with data_stream enabled exists and the indices have a @timestamp field. The indices become backing indices, hidden. Rehearse it on a copy first: clients that write with op_type=index start failing the moment the conversion completes.

Failure modes and anti-patterns

The template change did nothing. Symptom: new field mapped as text with a .keyword subfield instead of the keyword you defined. Confirm with GET <data-stream>/_mapping/field/<field>, then GET _data_stream/<name> to see the template name and GET _index_template/<that-name> to check the version. The write index predates the change. Fix: roll over, or apply a compatible mapping change to the data stream directly.

The wrong template won. Symptom: an index has none of your settings, or has settings from a template you did not expect. Confirm with POST _index_template/_simulate_index/<index-name> and read overlapping. Fix: narrow the patterns, adjust priority, and delete the stale template if it is unused.

A legacy template is still in play. Symptom: an older cluster has settings coming from nowhere for indices no composable template matches. Confirm with GET _template. Fix: port it to a composable template and delete it.

Writes to a data stream fail with an op type error. Symptom: bulk responses report that only op_type=create is allowed for data streams. A client or Logstash output is sending index actions. Fix: set the client to create. In Logstash's Elasticsearch output, enable data stream mode for that output, or set the action to create.

Documents rejected for a missing timestamp. Symptom: bulk errors about @timestamp. The producer sends timestamp or time instead. Fix in an ingest pipeline that renames or parses the field, set as the template's index.default_pipeline or index.final_pipeline; see ingest pipelines vs Logstash.

Accidental data stream. Symptom: an application that writes a single index named like logs-myapp-prod suddenly gets a data stream, because the built-in logs-*-* template matched. Confirm with GET _data_stream/<name>. Fix: rename the index to something outside the managed patterns, or create a higher-priority, narrowly matched template without data_stream. The cleanest fix is a name that does not collide.

One template per index. Symptom: hundreds of near-identical index templates, each differing in one field. Fix: move shared content into component templates and keep index templates thin.

Template with too many shards. Symptom: a quiet data stream with 5 primary shards rolled daily produces thousands of tiny shards. Fix: set shard count per data stream based on volume, and use size-based rollover. See shard sizing.

Runbook: changing a template safely

Use this when a mapping or setting change must reach production without surprises.

Read the current state. GET _index_template/<name> and every component template in its composed_of. Note the version.
Write the change in a component template, not the index template, unless the change is genuinely specific to this one template. Bump version.
Simulate. POST _index_template/_simulate with the proposed body, then POST _index_template/_simulate_index/<a-real-index-name> after applying it. Check overlapping is what you expect.
Check compatibility with live data. If you change a field type, search existing backing indices for that field: GET <data-stream>/_mapping/field/<field>. Decide whether you need a runtime field to bridge the old generations.
Apply the component template, then the index template.
Make it take effect. For compatible additions, PUT <data-stream>/_mapping. Otherwise POST <data-stream>/_rollover, or ?lazy=true for quiet streams.
Verify on the new write index: GET _data_stream/<name> for the newest backing index, then GET <that-index>/_mapping and GET <that-index>/_settings.
Watch the first hour of bulk responses from producers for mapping or op type errors.

And a checklist for reviewing a cluster's templates:

GET _template returns nothing, or only templates with an owner and a reason.
No composable template above priority 100 matches logs-*-*, metrics-*-* or synthetics-*-* broadly.
Every template has version and _meta.
Every data stream in GET _data_stream has a lifecycle (an ilm_policy, or data stream lifecycle retention).
No data stream receives routine updates by id.

Key takeaways

A template is applied once, when an index is created. Changing it affects the next index, not the current one; roll over to make it take effect.
Only one composable template applies to an index: the highest priority match. Component templates are how you compose; priority is how you choose.
_simulate_index and its overlapping field show which template wins and which ones lost, before you create anything.
Data streams are for append-only timestamped data: create operations, a @timestamp field, a retention policy. Routine updates by id belong in plain indices behind an alias.
Size-based rollover conditions keep shard size predictable; combine with age for retention granularity.
Do not override the built-in logs-*-* and metrics-*-* templates with broad patterns; use the @custom hooks.

How Sparced helps

Sparced reads cluster health, shard placement and ILM state in one pass, so a data stream whose backing indices are unhealthy shows up with the actual backing index and shard names rather than as a vague yellow status. When the cluster is red or yellow, it runs _cluster/allocation/explain for each stuck shard and prints the cluster's own explanation alongside a fix command with those names filled in. It also surfaces ILM state for managed indices, so a backing index that is not progressing through its policy appears as a finding with the evidence attached. It does not simulate or edit templates and writes nothing to the cluster; template design and changes remain your decision.