Elasticsearch ILM Troubleshooting: Fixing Stuck Policies

Index lifecycle management fails quietly. An index stuck in its hot phase does not turn the cluster yellow; it just keeps growing until a disk watermark or a shard size problem turns into an incident weeks later. The answer is to treat ILM as what it is, a per-index state machine that advances one step at a time on a timer, and to read its state with _ilm/explain before touching anything. Almost every stuck index falls into one of a handful of patterns: a rollover alias problem, a step waiting on shard allocation, a step that errored and needs a retry, phase timing that is measured from a different date than you assumed, or a policy change that the index has not picked up. This article covers how to tell them apart and fix each one.

The mental model: phases, actions, steps and a poll timer

An ILM policy is a set of phases (hot, warm, cold, frozen, delete), each containing actions (rollover, forcemerge, shrink, allocate, migrate, searchable_snapshot, delete and others). Each action is implemented as a sequence of steps. The index's position is always a triple: phase, action, step.

ILM does not run continuously. The master node runs ILM, and most step transitions are evaluated on a periodic poll whose interval is set by indices.lifecycle.poll_interval, which defaults to 10 minutes. Some steps chain immediately once the previous one completes, but anything that waits on a condition (rollover thresholds, shard allocation, min_age) is rechecked on that poll. Two consequences:

• An index that meets its rollover condition can take up to the poll interval to actually roll over. Rollover conditions are not hard ceilings.
• Fixing the cause of a wait does not produce an instant transition. Wait for the next poll before concluding the fix did not work.

There are three kinds of "not progressing":

Check the last one first, because it affects every index at once:

GET _ilm/status

{ "operation_mode": "RUNNING" }

STOPPING means ILM is waiting for in-flight operations to finish before stopping; STOPPED means nothing is advancing anywhere.

Reading _ilm/explain

_ilm/explain is the single most useful ILM API. It accepts an index name, a pattern, or a data stream, and two filters that matter during an incident: only_errors and only_managed.

GET logs-*/_ilm/explain?only_errors=true

A healthy index waiting for rollover looks like this:

{
  "indices": {
    ".ds-logs-app-default-2026.10.19-000031": {
      "index": ".ds-logs-app-default-2026.10.19-000031",
      "managed": true,
      "policy": "logs-app-30d",
      "index_creation_date_millis": 1792368000000,
      "time_since_index_creation": "9.6h",
      "lifecycle_date_millis": 1792368000000,
      "age": "9.6h",
      "phase": "hot",
      "phase_time_millis": 1792368001000,
      "action": "rollover",
      "action_time_millis": 1792368002000,
      "step": "check-rollover-ready",
      "step_time_millis": 1792368002000,
      "phase_execution": {
        "policy": "logs-app-30d",
        "phase_definition": {
          "min_age": "0ms",
          "actions": {
            "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" }
          }
        },
        "version": 4,
        "modified_date_in_millis": 1791000000000
      }
    }
  }
}

And an index in error:

{
  "index": "app-orders-000007",
  "managed": true,
  "policy": "orders-rollover",
  "phase": "hot",
  "action": "rollover",
  "step": "ERROR",
  "failed_step": "check-rollover-ready",
  "is_auto_retryable_error": true,
  "failed_step_retry_count": 412,
  "step_info": {
    "type": "illegal_argument_exception",
    "reason": "index.lifecycle.rollover_alias [app-orders] does not point to index [app-orders-000007]"
  }
}

The fields, in the order you should read them:

• managed: false means no policy is attached. Check index.lifecycle.name on the index or its template.
• step and failed_step: if step is ERROR, failed_step is where it broke.
• step_info.reason: the cluster's own explanation. Read it literally; it usually names the alias, setting or index at fault.
• is_auto_retryable_error and failed_step_retry_count: ILM retries certain transient errors automatically on each poll. A count in the hundreds means it has been failing for days, which a manual retry will not cure.
• age and lifecycle_date_millis: the age ILM uses for min_age comparisons. This is not always the creation date; see the timing section.
• phase_execution: the cached copy of the current phase definition, with the policy version it was taken from. This is what the index is actually executing, regardless of what the policy says now.
• step_time_millis: how long the index has been on this step. A wait step that has been current for days is the signal.

A useful sweep for a large cluster is to list everything not in the expected place. The _cat layer does not expose ILM, so filter the explain output with a script, or at least scan for indices whose step is not check-rollover-ready or complete in the hot phase.

Rollover failures, the most common class

Rollover is where most ILM errors happen, because it depends on configuration outside the policy.

Data streams

For a data stream, the rollover target is the data stream itself. There is no alias to configure, and the alias errors below do not apply. If a backing index is erroring on rollover, check that the data stream still exists, that the backing index is still its write index (an older generation that was the write index when it errored, and has since been superseded, may need a retry to move on), and that the template the data stream is bound to is still valid.

Plain indices with a write alias

For classic alias-based rollover, three conditions must all hold:

1. The index has index.lifecycle.rollover_alias set to the alias name.
2. That alias points to this index.
3. This index is the alias's write index (is_write_index: true), or the only index behind the alias.

The common errors map to those conditions:

The correct bootstrap for alias-based rollover:

PUT _index_template/orders
{
  "index_patterns": ["app-orders-*"],
  "template": {
    "settings": {
      "index.lifecycle.name": "orders-rollover",
      "index.lifecycle.rollover_alias": "app-orders"
    }
  },
  "priority": 200
}

PUT app-orders-000001
{
  "aliases": {
    "app-orders": { "is_write_index": true }
  }
}

The classic accident is an application writing to app-orders-000007 directly, or an earlier index being recreated by a stray write after deletion. Auto-creation by a write produces an index that matches the template, gets the policy and the rollover alias setting, but is not behind the alias. It then errors forever with "does not point to index". If that index should not exist, delete it or remove its policy:

POST app-orders-000007/_ilm/remove

Rollover that never happens

An index sitting in check-rollover-ready for a long time is not an error. It is waiting for a condition. Three reasons it may never be met:

• Conditions set too high for the volume. max_primary_shard_size: 50gb on a stream that writes 200 MB a day takes 250 days. Add max_age.
• Empty indices. In recent 8.x releases, ILM does not roll over an index that has no documents, even if max_age is met, to avoid a chain of empty indices: unless the policy sets its own document minimum, ILM adds an implicit minimum of one document. A quiet stream sits there until a document arrives. That is intended.
• index.lifecycle.indexing_complete set to true. This tells ILM the index has been rolled over externally, and the rollover action skips it. If it was set by mistake on an index that is still the alias's write index, the rollover step errors instead (see the table above) rather than rolling.

In recent 8.x releases rollover also has minimum conditions (min_age, min_docs, min_size, min_primary_shard_size, min_primary_shard_docs, inside the rollover action). These prevent rollover until the minimums are met even when a maximum is reached. If a policy has both, check that the minimums are not the thing holding it.

Steps waiting on allocation

Several actions need shards to move before ILM can continue, and they wait rather than error when shards cannot move.

The explain output for a waiting migrate step shows a message in step_info saying how many shards are still waiting to be moved to the target tier, and the tier preference it is using. The question then becomes an allocation question, not an ILM one:

GET _cluster/allocation/explain
{
  "index": ".ds-logs-app-default-2026.10.01-000020",
  "shard": 0,
  "primary": true
}

Typical answers: there are no nodes with the data_warm role at all (the cluster has no warm tier, so the index should not be told to migrate), the warm nodes are past the high disk watermark (90% by default), or a filter in the policy's allocate action names an attribute no node has. The deciders and their messages are covered in unassigned shards and the disk side in disk watermarks.

If a cluster genuinely has no warm tier and the policy has a warm phase, either remove the phase or disable the implicit migration in it:

"warm": {
  "min_age": "7d",
  "actions": {
    "migrate": { "enabled": false },
    "forcemerge": { "max_num_segments": 1 }
  }
}

Shrink deserves a specific warning. It first pins a copy of every shard to a single node, which requires that node to hold a full copy of the index: on a 500 GB index, 500 GB on one node. The shrunken index is then built from hard links where the file system supports them, so it does not necessarily double the space, but if hard links are not possible the segments are copied and you need up to twice that. If no node has the headroom under the high watermark, the shrink waits. On recent versions, ILM can choose a different node if the first choice cannot take the shards, but it cannot invent disk. Shrink is also usually unnecessary if rollover is size-based and the template sets a sensible shard count.

Phase timing: what min_age is measured from

min_age on a phase is compared against the index's lifecycle date, which ILM sets as follows:

1. If index.lifecycle.origination_date is set, that value.
2. Otherwise, if the index has been rolled over, the time of the rollover.
3. Otherwise, the index creation date.

Rule 2 is the one that surprises people. A policy with warm.min_age: 7d on a rolled-over index means 7 days after that index stopped being the write index, not 7 days after its first document. Worked example with daily rollover, warm: 7d, delete: 30d:

1. Index created Monday 00:00, written to all day, rolled over Tuesday 00:00. Lifecycle date becomes Tuesday 00:00.
2. Warm begins 7 days later: the following Tuesday 00:00 plus up to one poll interval.
3. Delete begins 30 days after Tuesday 00:00.
4. The oldest document in it was written Monday 00:00, so it is deleted at about 31 days of age.

Retention therefore equals the delete min_age plus up to one rollover period plus one poll interval. With max_age: 7d rollover and delete.min_age: 30d, documents can survive about 37 days. If a compliance rule says "no more than 30 days", set the delete phase to 30 days minus the rollover period.

The opposite surprise happens when you attach a policy to old data. After a reindex, the new index's creation date is today, so nothing moves for weeks even though the documents are months old. Or the data is in indices created months ago, and the moment you attach the policy everything leaps to delete. Set the lifecycle date explicitly:

PUT legacy-logs-2026.03/_settings
{
  "index.lifecycle.origination_date": 1772323200000
}

Alternatively, index.lifecycle.parse_origination_date: true tells ILM to parse the date from an index name of the form name-yyyy.MM.dd or name-yyyy.MM.dd-000001. If the name does not match that form, the index fails at creation, so test it first.

Retrying, moving and removing

Once the cause is fixed, retry the failed step:

POST app-orders-000007/_ilm/retry

Retry puts the index back on the failed step. It only works for indices in the ERROR step; on a waiting index it returns an error. If the cause is not fixed, the index goes straight back to ERROR on the next attempt, with failed_step_retry_count incremented.

When an index needs to skip a step, or be pushed to a different phase, use the move API. It requires the current step exactly, as a guard against moving an index that has already progressed:

POST _ilm/move/.ds-logs-app-default-2026.10.01-000020
{
  "current_step": {
    "phase": "warm",
    "action": "shrink",
    "name": "check-shrunken-shards-allocated"
  },
  "next_step": {
    "phase": "cold"
  }
}

Copy current_step from _ilm/explain output, not from memory; a mismatch makes the call fail. In next_step you may give just a phase, or a phase and action, and ILM starts at the first step of it, or you can name a specific step. Moving skips everything between the two positions, so moving an index from hot directly to the delete phase can delete it within minutes.

Removing the policy (_ilm/remove) detaches ILM entirely. The index keeps whatever state it reached. Note that remove clears index.lifecycle.name and the related lifecycle settings on the index. Templates do not re-apply to existing indices, so re-attaching means setting index.lifecycle.name yourself, and the index then starts at the beginning of the policy, with phase timing computed from its lifecycle date.

Updating a policy on indices already in flight

When you change a policy, indices currently executing it do not immediately switch to the new definition of their current phase. Each index cached the phase definition when it entered the phase, visible as phase_execution in the explain output. The general rule, as Elastic documents it, is that a changed phase definition takes effect when the index next enters a phase. In recent versions ILM also refreshes the cached definition for indices in the changed phase when it can do so safely, for example when the step the index is on still exists in the new definition. Do not depend on either behavior from memory; after a policy update, check phase_execution.version against the policy's version for a sample of indices:

GET _ilm/policy/logs-app-30d

{
  "logs-app-30d": {
    "version": 5,
    "modified_date": "2026-10-21T09:12:00.000Z",
    "policy": { "phases": { } },
    "in_use_by": {
      "indices": [".ds-logs-app-default-2026.10.19-000031"],
      "data_streams": ["logs-app-default"],
      "composable_templates": ["acme-logs-app"]
    }
  }
}

in_use_by tells you the blast radius of an edit before you make it: every index, data stream and template that references this policy.

Practical consequences:

• Changing a rollover threshold on the hot phase may not affect the current write index until it is refreshed or rolled over. A manual rollover gets you onto the new definition at once.
• Deleting a phase that some indices are already in leaves those indices finishing the cached version of it. They then move to the next phase that exists.
• Shortening retention in the delete phase affects indices that have not yet entered delete. Indices already in delete are deleted anyway.
• Switching an index to a different policy by changing index.lifecycle.name: the index finishes its current phase using the cached definition, then follows the new policy. If the new policy lacks the phase or step the index is in, check explain immediately afterwards.

For large changes, a new policy name is often clearer than an in-place edit. Point the template at the new policy and roll over; new indices follow the new rules, old indices age out under the old ones, and nothing is ambiguous.

Failure modes and anti-patterns

ILM stopped and nobody noticed. Symptom: no index has advanced in days. Confirm: GET _ilm/status returns STOPPED. Cause: a maintenance window ran _ilm/stop and the start was forgotten. Fix: POST _ilm/start, and add ILM status to your monitoring.

Delete phase waiting on a snapshot. Symptom: delete phase indices stuck on wait-for-snapshot. The wait_for_snapshot action waits until an SLM policy has taken a snapshot that includes the index. Confirm: GET _slm/policy/<name> for last_failure. Fix: repair SLM; see snapshot and restore.

Searchable snapshot action failing. Symptom: cold or frozen phase in ERROR with a repository exception in step_info. Confirm: POST _snapshot/<repo>/_verify. Fix: restore repository access, then retry.

Force merge on the hot tier. Symptom: hot nodes show heavy merge I/O and indexing latency rises daily at rollover. The policy force merges in the hot phase right after rollover, competing with live indexing. Fix: move forcemerge to warm, or accept it knowingly. See indexing performance.

Policy attached to the wrong indices. Symptom: an unexpected index was deleted. The template's patterns were too broad and the policy rode along. Confirm: in_use_by on the policy. Fix: narrow the template patterns; for the future, use _simulate_index before changing templates. See index templates and data streams.

Lowering the poll interval in production. Setting indices.lifecycle.poll_interval to a few seconds is useful in a test cluster to watch a policy run. In production it adds master load for little gain; leave it at its default unless you have a measured reason.

Runbook: an index is stuck

Follow in order. Most incidents end at step 4.

1. Is ILM running? GET _ilm/status. If not RUNNING, POST _ilm/start and wait one poll interval.
2. Find the errors. GET */_ilm/explain?only_errors=true&filter_path=indices.*.step_info,indices.*.failed_step,indices.*.policy. Group by step_info.reason; usually one cause explains many indices.
3. Read the reason literally. It names the alias, setting, repository or index. Map it to the rollover table above, or to allocation, repository or settings causes.
4. Fix the cause, then retry. POST <index>/_ilm/retry. Wait one poll. Re-run explain.
5. No error but no progress? Look at step and step_time_millis: - check-rollover-ready: compare the rollover conditions to the index's actual size and age (GET _cat/indices/<index>?v&h=index,pri.store.size,docs.count,creation.date.string). Check indexing_complete. - an allocation or migrate step: run _cluster/allocation/explain on one of its shards. - wait-for-snapshot: check SLM.
6. Check timing. Compare age and lifecycle_date_millis to what you expected. If the date is wrong, set index.lifecycle.origination_date.
7. Check the cached definition. Compare phase_execution.version with GET _ilm/policy/<name>. If the index is executing an old definition and that matters, roll over or move it.
8. Last resort: move. POST _ilm/move/<index> with current_step copied from explain. Record what you skipped, because a skipped force merge or shrink does not happen later.

Key takeaways

• ILM is a per-index state machine evaluated on a poll, every 10 minutes by default. Transitions are not instant.
• _ilm/explain tells you the phase, action and step, the reason for any error, the age ILM is using, and the cached phase definition. Read it before changing anything.
• Most errors are rollover alias problems for plain indices: the alias setting, the alias membership, and the write index flag must all line up. Data streams avoid that class entirely.
• min_age is measured from rollover for rolled-over indices, so real retention is the delete age plus a rollover period plus a poll.
• Waiting steps on migrate, allocate and shrink are allocation problems; diagnose them with the allocation explain API.
• Policy edits may not reach in-flight indices immediately; check phase_execution.version, and consider a new policy plus rollover for large changes.

How Sparced helps

Sparced reads ILM state alongside cluster health, shards and allocation in a single pass, so indices in an ILM error step appear as findings with the index names and the cluster's own step_info reason attached. Because it reads allocation and disk in the same pass, an ILM step waiting on shard movement can be seen next to the allocation state that is blocking it, and when the cluster is red or yellow it runs _cluster/allocation/explain for each stuck shard. Each finding comes with a fix command with the index names filled in, and a check to confirm the fix took. It does not retry, move or edit anything itself; it writes nothing to the cluster.