Cutting Elasticsearch Storage Costs: logsdb, TSDS and Codecs

Elasticsearch storage bills grow for boring reasons: every string mapped twice by dynamic mapping, _source stored in the fast codec, replicas on data nobody has queried in a month, and retention decided once and never revisited. The good news is that most of the waste is measurable per field and per index, and the fixes are mechanical. The approach in this article is: measure first with the disk usage and field usage APIs, then apply the levers in order of payoff and risk: retention and tiering (replica and copy count), the index mode (logsdb for logs, TSDS for metrics), the source and codec choices, mapping trims, and finally force merge. Each section explains what the lever changes inside Lucene, so you can predict its effect instead of hoping.

The mental model: where the bytes go

An index's disk footprint is the sum of Lucene structures per field, times the number of shard copies, plus waste from deleted documents not yet merged away.

Four facts follow:

1. _source is often the largest single item, because it holds every field again, as text.
2. index.codec compresses stored fields only. Switching to best_compression shrinks _source; it does nothing to doc values or postings.
3. A dynamically mapped string is stored up to three times: in _source, as a text field (inverted index plus norms), and as the .keyword subfield that default dynamic mapping adds (inverted index plus doc values).
4. Deleted and updated documents still occupy space until a merge rewrites their segment. An index with heavy updates can carry a large share of dead documents.

Replicas multiply all of it. One replica doubles the cluster's storage for an index; that is often the largest single lever, and it is a decision about availability, not compression.

Measure before you change anything

Index-level sizes

GET _cat/indices?v&h=index,pri,rep,docs.count,docs.deleted,pri.store.size,store.size&s=store.size:desc

pri.store.size is one copy; store.size includes replicas. A high docs.deleted relative to docs.count points to merge waste.

Per-field disk usage

The disk usage API analyzes an index and reports bytes per field and per structure. It is expensive (it reads the index) and has been a technical preview, so run it on one representative index, off-peak:

POST logs-app-2026.10.14/_disk_usage?run_expensive_tasks=true

A trimmed response:

{
  "logs-app-2026.10.14": {
    "store_size_in_bytes": 52613874012,
    "all_fields": {
      "total_in_bytes": 52480112355,
      "inverted_index": { "total_in_bytes": 15120998712 },
      "stored_fields_in_bytes": 27410331809,
      "doc_values_in_bytes": 8831005611,
      "points_in_bytes": 701255980,
      "norms_in_bytes": 416520243
    },
    "fields": {
      "_source": { "total_in_bytes": 27100994310 },
      "message": { "total_in_bytes": 9204003311 },
      "message.keyword": { "total_in_bytes": 3110442981 }
    }
  }
}

Read it as a ranked list of candidates. In this example, stored fields (mostly _source) are 52% of the index, and message.keyword, a subfield nobody aggregates on, costs 3.1 GB per day per copy.

Which fields are actually used

The field usage stats API (also a technical preview) reports, per shard, which fields have been accessed by queries and how (inverted index, doc values, stored fields, norms):

GET logs-app-2026.10.14/_field_usage_stats

Counters are per shard and since the shard was last started, so collect them over a representative period and across shards before concluding a field is unused. A field with zero inverted index and doc values access over weeks is a candidate for index: false, doc_values: false or removal.

The levers, in order of payoff

"Typical effect" is deliberately qualitative. Compression depends on your data; measure on one index with _disk_usage before and after, and plan from your numbers.

Retention, replicas and tiers

Before compressing bytes, check how many copies of them you keep and on what hardware.

Worked example, with stated assumptions: 200 GB per day of primary storage after indexing, 30 days of retention, one replica.

• Everything on hot nodes with one replica: 200 * 30 * 2 = 12,000 GB of hot-tier disk.
• Hot for 7 days with one replica, then cold as fully mounted searchable snapshots with no replicas: hot 200 * 7 * 2 = 2,800 GB; cold 200 * 23 * 1 = 4,600 GB local disk, with redundancy provided by the snapshot repository. Total local disk 7,400 GB, much of it on cheaper nodes.
• Hot for 7 days, then frozen (partially mounted): hot 2,800 GB; frozen nodes hold only a cache, with the full 23 days in the object store.

Fully mounted indices on the cold tier can run without replicas because a lost shard is recovered from the snapshot rather than from a replica, at the cost of slower recovery. Partially mounted indices on the frozen tier keep only a shared cache on local disk; on nodes with only the frozen role, xpack.searchable.snapshot.shared_cache.size defaults to 90% of the disk. Searchable snapshots require an appropriate subscription. The full model, including when frozen query latency is acceptable, is in data tiers and ILM, and the sizing arithmetic is in capacity planning for logging.

Downsampling is the retention lever for metrics: keep raw resolution for a few days, then roll up to 5-minute or 1-hour intervals with the ILM downsample action on a TSDS (below).

Index modes: logsdb and TSDS

The biggest structural savings for observability data come from index modes that change how Lucene lays out the data.

logsdb for logs

PUT _index_template/logs-app
{
  "index_patterns": ["logs-app-*"],
  "data_stream": {},
  "priority": 300,
  "template": {
    "settings": {
      "index.mode": "logsdb"
    }
  }
}

What the mode does:

• Index sorting by host.name and @timestamp by default, so similar documents sit next to each other and compress far better.
• Synthetic _source, so the original JSON is not stored; it is rebuilt from doc values and stored fields when needed.
• Specialized codecs for doc values and stored fields suited to sorted log data.

In recent releases, logsdb is generally available and Elastic has made it the default for the built-in logs-*-* data streams in 9.x. Check what a data stream actually uses rather than assuming:

GET _data_stream/logs-app-default

The response includes the index mode of the data stream. Synthetic source under logsdb has been tied to the subscription level in recent releases, with logsdb falling back to stored source where the license does not cover it; confirm on your version and license before you put savings in a budget.

TSDS for metrics

A time series data stream is a data stream with index.mode: time_series. You declare which fields identify a series (dimensions) and which are measurements (metrics):

PUT _index_template/metrics-node
{
  "index_patterns": ["metrics-node-*"],
  "data_stream": {},
  "priority": 300,
  "template": {
    "settings": {
      "index.mode": "time_series",
      "index.routing_path": ["host.name", "metric.set"]
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "host.name":  { "type": "keyword", "time_series_dimension": true },
        "metric.set": { "type": "keyword", "time_series_dimension": true },
        "cpu.pct":    { "type": "double",  "time_series_metric": "gauge" },
        "net.bytes":  { "type": "long",    "time_series_metric": "counter" }
      }
    }
  }
}

TSDS routes documents of the same series to the same shard, sorts by series and time, uses synthetic source, and builds the document _id from the dimensions and timestamp. The consequences you must design for:

• Duplicates are rejected. Two documents with the same dimensions and timestamp conflict. That is usually correct for metrics and surprising for anything else.
• Each backing index accepts a time window. Documents with a timestamp outside the accepted range for the current backing indices are rejected, so very late or future-dated data needs attention.
• Downsampling becomes available:

{
  "policy": {
    "phases": {
      "hot":  { "actions": { "rollover": { "max_age": "1d", "max_primary_shard_size": "50gb" } } },
      "warm": { "min_age": "7d", "actions": { "downsample": { "fixed_interval": "1h" } } },
      "delete": { "min_age": "395d", "actions": { "delete": {} } }
    }
  }
}

Worked arithmetic for downsampling: a series reported every 10 seconds produces 360 points per hour. Downsampled to 1 hour, it keeps one aggregated document per hour (holding min, max, sum and value count for gauges), a 360-to-1 reduction in document count for that period. The byte saving is smaller than the document ratio because each downsampled document carries several aggregates, but over a year of retention it dominates every other lever. See data streams and templates.

Source and codec

Synthetic _source

Outside logsdb and TSDS you can enable synthetic source on an ordinary index. In recent releases it is an index setting:

PUT orders-v3
{
  "settings": {
    "index.mapping.source.mode": "synthetic"
  }
}

Older 8.x releases configured it in the mapping as "_source": { "mode": "synthetic" }; that form has been deprecated in favor of the setting. Elastic's documentation marks synthetic source as a feature that requires a subscription, so check your license before planning around it.

What you give up, and must test:

• The returned document is reconstructed, not the original bytes. Field order can change, and arrays of some types come back sorted and deduplicated.
• Every field must be reconstructable, which means relying on doc values or stored fields. For fields with neither, and for values dropped by ignore_above or ignore_malformed, the original value is kept as stored source, which reduces the saving.
• Reconstruction costs CPU when you fetch documents, so search hits and reindex do slightly more work.

It is a good trade for high-volume, rarely fetched data. It is a poor one for indices where the application needs the exact original JSON back.

best_compression and the codec

POST logs-app-2026.10.14/_close

PUT logs-app-2026.10.14/_settings
{
  "index": { "codec": "best_compression" }
}

POST logs-app-2026.10.14/_open

index.codec is a static setting: set it in the index template for new indices, or, for an existing index, close it, change it and reopen it as above (the index is unavailable while closed). Existing segments keep their old compression until they are rewritten, so a force merge (or ILM's forcemerge action with "index_codec": "best_compression" in the warm phase) is what actually shrinks old data.

The default codec uses LZ4 for stored fields. best_compression uses a higher-ratio algorithm: zstd in current releases (older releases used DEFLATE). The trade is CPU: compression during indexing and merging, and decompression when fetching stored fields. For data past its first days, that trade almost always pays. For the hottest indices with heavy fetch traffic, measure.

Remember the scope: the codec touches stored fields only. If _disk_usage shows stored fields are 20% of an index, the best case for this lever is a share of that 20%.

Mapping trims

Mapping decides which structures exist for each field. Trims save space without touching what is stored in _source, but most need a reindex for existing data (see zero-downtime reindexing), so put them in the template and let them take effect at the next rollover.

The most common single win is stopping default dynamic mapping from creating a text field and a .keyword subfield for every string. A dynamic template that maps strings to keyword only (or to match_only_text for a known message field) prevents it:

PUT _component_template/strings-as-keyword
{
  "template": {
    "mappings": {
      "dynamic_templates": [
        {
          "strings_as_keyword": {
            "match_mapping_type": "string",
            "mapping": { "type": "keyword", "ignore_above": 1024 }
          }
        }
      ],
      "properties": {
        "message": { "type": "match_only_text" }
      }
    }
  }
}

Numbers: choose the smallest type that holds the values. scaled_float with a scaling_factor stores prices or percentages as scaled longs. Identifiers that are never range-queried are often better as keyword. See mapping design for the full set.

Index sorting (index.sort.field and index.sort.order, fixed at index creation) can also improve compression for any index, by putting similar values next to each other. It adds indexing cost; logsdb and TSDS apply it for you.

Force merge and deleted documents

Lucene merges segments continuously, and the merge policy reclaims deleted documents as it goes. Two cases call for an explicit force merge:

• Read-only indices (a rolled-over time-series index, a finished reindex). Merging to one segment per shard reduces per-segment overhead and improves compression a little.
• Indices with many deleted documents that are no longer written to.

POST logs-app-2026.10.14/_forcemerge?max_num_segments=1

Rules:

• Only force merge indices that no longer receive writes. The default merge policy does not merge segments larger than about 5 GB (index.merge.policy.max_merged_segment), so a huge force-merged segment in an index that keeps receiving updates accumulates deletes that are not reclaimed until a large share of it is deleted.
• It is IO and CPU heavy and needs free disk for the new segments while the old ones still exist. Watch the disk watermarks (low 85%, high 90%, flood stage 95% by default). See disk watermarks.
• Let ILM do it. The warm phase forcemerge action runs it after rollover, and can apply best_compression in the same step.
• Do it before a snapshot that will back a searchable snapshot index, so the snapshot holds the compact form.

Failure modes and anti-patterns

Runbook: a storage reduction pass

1. Rank indices by size: _cat/indices sorted by store.size. Group by data stream or pattern; the top few patterns are usually most of the bill.
2. Check copies: replicas per pattern, and how many days sit on the hot tier. Compare with the access pattern. This is a conversation with the data owners, not a setting change.
3. Pick one representative index per top pattern and run _disk_usage on it off-peak. Record the per-structure totals and the top 20 fields.
4. Collect field usage stats over at least a week for the same pattern.
5. Choose the index mode: logsdb for logs, TSDS for metrics with clear dimensions, standard otherwise. Confirm licensing for synthetic source.
6. Write the template changes: index mode, codec, dynamic templates, field trims, removed fields in the ingest pipeline.
7. Test on a copy: reindex one day's data into a test index with the new template, run _disk_usage on it, and compare. Run the dashboards and queries that depend on it.
8. Roll out by rollover: apply the template and roll over; the change applies to new backing indices.
9. Add or adjust ILM: warm phase forcemerge with best_compression, cold or frozen phases, downsampling for TSDS, delete phase matching the agreed retention.
10. Verify a week later: compare daily primary size per pattern before and after, and check disk headroom on each tier.

Key takeaways

• Measure per field with _disk_usage and per use with _field_usage_stats before changing anything; plan from your own before-and-after numbers.
• Copies and retention usually beat compression: replicas, tiers and searchable snapshots change storage by whole multiples.
• logsdb (logs) and TSDS (metrics) are the largest structural levers; they sort the data, use synthetic source and specialized codecs, and TSDS unlocks downsampling.
• best_compression affects stored fields only and takes effect on rewritten segments; pair it with a force merge in the warm phase.
• Stop dynamic mapping from storing every string as text plus .keyword; trim structures each field does not need.
• Force merge only indices that no longer receive writes, with disk headroom for the rewrite.

How Sparced helps

Sparced reads index, shard, node and ILM state in one pass, so a storage review starts from facts: disk usage per node against the low, high and flood stage watermarks, shard counts and sizes per index, ILM policies with any stuck steps, and the state of snapshot repositories and SLM that searchable snapshot tiers depend on. Each finding names the indices, shards and nodes involved, gives a fix command with those names filled in, and a check to confirm the fix. It does not run _disk_usage, change codecs or mappings, or force merge anything, because it writes nothing to the cluster; those changes stay your decision.