Elasticsearch Relevance Tuning: BM25, Boosts and rank_eval

Most relevance work in Elasticsearch fails the same way: someone sees a bad result, adds a boost, the bad result moves, three other queries quietly get worse, and nobody notices for a month. The fix is not a cleverer boost. It is a loop: read why a document scored the way it did (the explain API), change one thing with a known mechanical effect (BM25 parameters, field weights, the multi_match type, a function_score or rank_feature signal, synonyms), and measure the change against a fixed set of judged queries (the rank_eval API). This article gives you the scoring model, the levers in the order you should reach for them, and the measurement harness that stops relevance tuning from becoming folklore.

The mental model: what a score actually is

Every full-text clause in Elasticsearch is scored by the field's similarity, which is BM25 unless you configure otherwise. The score of a document is assembled bottom-up:

1. Each term in each matching field produces a BM25 score.
2. Query structure combines those: a bool query sums the scores of matching must and should clauses, a dis_max takes the best clause plus a tie-breaker fraction of the others, and filter and must_not clauses contribute nothing to the score.
3. Boosts multiply clause scores.
4. Wrappers such as function_score or script_score then reshape the final number.

Three facts about that pipeline drive most surprises:

• Statistics are per shard. BM25 uses document frequency (how many documents contain the term) and average field length. By default these come from the shard that holds the document, not the whole index. On a small index with many shards, the same term can have noticeably different IDF on different shards, so identical documents score differently. The dfs_query_then_fetch search type gathers global term statistics first at the cost of an extra round trip. On large, evenly distributed indices the per-shard statistics converge and the difference stops mattering, which is why relevance bugs often reproduce in a small test index and vanish in production, or the other way round.
• Scores are not comparable across queries. A score of 14.2 on one query and 7.1 on another tells you nothing. Never threshold on raw BM25 scores, and never assume that a boost of 2 means "twice as important" across fields with very different term statistics.
• Field length is stored lossily. Lucene stores the field length norm in a single byte, so the dl value you see in explain output is an approximation. Do not chase tiny score differences between two documents of similar length.

BM25 in numbers

Lucene's BM25 for a single term in a single field is:

score = boost * idf * tf

idf = ln(1 + (N - n + 0.5) / (n + 0.5))
tf  = freq / (freq + k1 * (1 - b + b * dl / avgdl))

where N is the number of documents with the field on the shard, n the number containing the term, freq the term frequency in this field, dl the field length and avgdl the average length. The defaults are k1 = 1.2 and b = 0.75. In explain output the default boost appears as 2.2, because Lucene folds the (k1 + 1) factor into the boost.

A worked example on a shard of 1,000,000 documents:

A rare term is worth about four times a common one before anything else happens. Now term frequency, for a document whose field length equals the average (so the length factor is exactly 1):

• freq = 1: tf = 1 / (1 + 1.2) = 0.455
• freq = 3: tf = 3 / (3 + 1.2) = 0.714
• freq = 10: tf = 10 / (10 + 1.2) = 0.893

That saturation curve is the point of BM25: the tenth occurrence adds far less than the second. k1 controls how fast it saturates. Lower k1 (say 0.5) makes one occurrence almost as good as ten; higher k1 rewards repetition.

Length normalization: the same freq = 1 in a field twice the average length, with b = 0.75:

• length factor = 1 - 0.75 + 0.75 * 2 = 1.75
• tf = 1 / (1 + 1.2 * 1.75) = 1 / 3.1 = 0.323

So the long document's match is worth about 71% of the average-length document's. b controls how hard length is punished. b = 0 disables length normalization; b = 1 applies it fully.

When to change k1 and b

Changing BM25 parameters is a legitimate lever but a blunt one, because it applies to every query against that field. Reasonable cases:

A custom similarity is defined in index settings and referenced from the mapping:

PUT products-v2
{
  "settings": {
    "index": {
      "similarity": {
        "title_bm25": { "type": "BM25", "k1": 0.9, "b": 0.4 }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": { "type": "text", "similarity": "title_bm25" },
      "body":  { "type": "text" }
    }
  }
}

Treat a similarity change as an index rebuild: the similarity mapping parameter of an existing field cannot be changed in place, so plan it with a reindex and an alias swap (see zero-downtime reindexing). Also note that norms: false on a field removes length normalization entirely and saves space; keyword fields have no norms by default.

Reading a score with the explain API

Before you change anything, find out why the wrong document won. There are two ways:

GET products/_explain/sku-48113
{
  "query": {
    "multi_match": {
      "query": "wireless noise cancelling headphones",
      "fields": ["title^3", "brand^2", "description"]
    }
  }
}

or add "explain": true to a normal search to get an _explanation tree on every hit. A trimmed term-level node looks like this:

{
  "value": 9.84,
  "description": "weight(title:wireless in 2210) [PerFieldSimilarity], result of:",
  "details": [
    {
      "value": 9.84,
      "description": "score(freq=1.0), computed as boost * idf * tf from:",
      "details": [
        { "value": 6.6, "description": "boost" },
        { "value": 3.12, "description": "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:" },
        { "value": 0.478, "description": "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:" }
      ]
    }
  ]
}

Here the boost of 6.6 is the field boost of 3 multiplied by 2.2. What to look for when reading these trees:

• Which field won. With best_fields, only the best field counts (plus tie-breaker). If description keeps winning over title, the description is long and repetitive or the title boost is too low.
• An idf that is unexpectedly high. A term that is rare on this shard (a typo, a code, a rare brand) can dominate everything. That is BM25 working as designed; the fix is usually query structure, not k1.
• Scores being summed where you expected a max. most_fields and bool should add up; three weak matches can beat one strong one.
• Function scores multiplying something you did not expect. The function_score node shows each function's value separately.

Explain is expensive. Use it on one document or a small result page, never in production traffic.

Field weights and multi_match types

The multi_match type is a bigger decision than any boost, because it changes how field scores combine.

Two related queries are worth knowing:

• combined_fields scores several text fields as if they were one field, following the BM25F idea. It requires the fields to share the same search analyzer, and it is often the cleanest answer to "words are spread across title, subtitle and tags".
• dis_max gives you the best_fields behavior with arbitrary sub-queries, when you need different query types per field.

A typical e-commerce baseline:

GET products/_search
{
  "query": {
    "bool": {
      "must": {
        "multi_match": {
          "query": "sony wireless headphones",
          "type": "cross_fields",
          "fields": ["title^3", "brand^2", "category", "description^0.5"],
          "operator": "and"
        }
      },
      "should": [
        {
          "match_phrase": {
            "title": { "query": "sony wireless headphones", "slop": 2, "boost": 2 }
          }
        }
      ],
      "filter": [ { "term": { "in_stock": true } } ]
    }
  }
}

The must clause decides what matches. The should phrase clause only adds score, rewarding documents where the words appear close together in the title. The filter restricts without scoring and is cacheable.

Choosing boost values

A rule of thumb that holds up: start with weights that reflect field importance in a small range (title 2 to 4, supporting fields 1, long bodies 0.5 to 1), then move them one at a time while watching the metric. Because BM25 values differ by field (titles are short, so their tf is high and their idf distribution differs), a boost of 3 on title does not mean title is three times as important. Only measurement tells you what a boost does.

minimum_should_match is a recall-versus-precision dial that often matters more than boosts. "minimum_should_match": "75%" on a four-term query requires three terms. True index-time boosting is gone: since 5.0 a boost set in the mapping is applied at query time anyway, and Elastic discourages setting it there. Keep all boosting in the query, where it can be changed without touching the mapping.

Business signals: function_score and rank_feature

Text relevance alone rarely matches what users want. Popularity, recency, margin, rating and availability matter. There are two mechanisms, and the second is usually better.

function_score

function_score wraps a query and modifies its score with one or more functions: weight, field_value_factor, script_score, random_score, and the decay functions gauss, exp and linear.

GET products/_search
{
  "query": {
    "function_score": {
      "query": { "multi_match": { "query": "headphones", "fields": ["title^3", "description"] } },
      "functions": [
        {
          "field_value_factor": {
            "field": "sales_30d",
            "modifier": "log1p",
            "factor": 1,
            "missing": 0
          }
        },
        {
          "gauss": {
            "release_date": { "origin": "now", "scale": "180d", "offset": "30d", "decay": 0.5 }
          }
        }
      ],
      "score_mode": "multiply",
      "boost_mode": "multiply",
      "max_boost": 10
    }
  }
}

The two knobs that confuse people:

• score_mode combines the functions with each other (default multiply; alternatives include sum, avg, first, max, min).
• boost_mode combines the function result with the query score (default multiply; alternatives include sum, replace, avg, max, min).

Worked arithmetic for the decay above: a product released 30 days ago sits inside the offset and gets 1.0. One released 210 days ago is one scale (180d) beyond the offset and gets exactly the decay value, 0.5. Further out it keeps falling along the Gaussian curve.

For field_value_factor with log1p: a product with 0 sales gets log10(1 + 0) = 0, which with multiply zeroes the whole score. That is the single most common function_score bug. Either use boost_mode: sum, add a constant weight function, or use a modifier like log2p (log10(2 + x)), which gives 0.30 for zero sales instead of 0. Note also that plain log of a value of 0 is undefined, which is why the 1p and 2p variants exist.

rank_feature

For static, per-document signals (popularity, page rank, quality score), the rank_feature field and query are usually the better tool:

PUT products-v2/_mapping
{
  "properties": {
    "popularity": { "type": "rank_feature" },
    "return_rate": { "type": "rank_feature", "positive_score_impact": false }
  }
}

GET products-v2/_search
{
  "query": {
    "bool": {
      "must": { "match": { "title": "headphones" } },
      "should": [
        { "rank_feature": { "field": "popularity", "saturation": { "pivot": 500 } } },
        { "rank_feature": { "field": "return_rate", "boost": 0.5 } }
      ]
    }
  }
}

Why it is better: the feature contributes an additive score in a should clause, so it never zeroes the text score; the functions are bounded (with saturation, a value equal to the pivot scores 0.5 and the score approaches 1 as the value grows); and because the scoring is known in advance, Lucene can skip non-competitive documents efficiently when you do not need an exact total hit count. The available functions are saturation, log, sigmoid and linear. If you omit the function and the pivot, Elasticsearch uses saturation with a pivot derived from the index's own feature values (an approximate geometric mean). positive_score_impact: false inverts the feature for signals where lower is better.

Rank features must be positive numbers, and the field is not usable for sorting or aggregations. Keep the raw value in a separate numeric field if you need those.

Synonyms done properly: search-time graphs

Synonyms are the most common relevance fix and the most common source of relevance bugs. The modern pattern is:

• Apply synonyms at search time only, with the synonym_graph token filter in a search analyzer.
• Keep the index analyzer free of synonyms.
• Manage the synonym list so it can be updated without reindexing.

Why search time: index-time synonyms bake the list into every document, so any change means a full reindex, and they distort term statistics. Why synonym_graph: multi-word synonyms ("ny" to "new york") produce token graphs, and only the graph filter represents them correctly at query time. Using synonym_graph at index time does not help, because the index cannot store a graph and the structure is flattened.

PUT products-v3
{
  "settings": {
    "analysis": {
      "filter": {
        "product_synonyms": {
          "type": "synonym_graph",
          "synonyms_set": "product-synonyms",
          "updateable": true
        }
      },
      "analyzer": {
        "text_index":  { "tokenizer": "standard", "filter": ["lowercase"] },
        "text_search": { "tokenizer": "standard", "filter": ["lowercase", "product_synonyms"] }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": { "type": "text", "analyzer": "text_index", "search_analyzer": "text_search" }
    }
  }
}

The synonyms_set parameter references a set stored with the synonyms API, available in recent 8.x releases:

PUT _synonyms/product-synonyms
{
  "synonyms_set": [
    { "id": "tv",   "synonyms": "tv, television" },
    { "id": "hp",   "synonyms": "headphones, headset, earphones" },
    { "id": "nyc",  "synonyms": "ny, nyc => new york" }
  ]
}

Updating the set through the API reloads the search analyzers that use it. If you instead use an inline list or a synonyms_path file with updateable: true, you call POST products-v3/_reload_search_analyzers after changing the file on every node. updateable: true is only allowed on filters used in search analyzers, which is another reason the search-time pattern is the right one.

Synonym rules of thumb:

• Equivalence lists (a, b, c) expand in all directions. Explicit mappings (a, b => c) replace. Use explicit mappings for abbreviations and misspellings, equivalence for true synonyms.
• Keep the synonym filter after lowercasing and before stemming, or the rules must be written in stemmed form.
• Watch match queries with multi-word synonyms: by default (auto_generate_synonyms_phrase_query: true) a multi-word synonym becomes a phrase query, which is usually what you want ("new york" as a phrase, not "new" or "york").
• Every new synonym is a relevance change. Run it through the rank_eval set like any other change.

Measuring with the rank_eval API

Without measurement, you cannot tell an improvement from a trade. The _rank_eval API runs a set of queries and computes an information retrieval metric against documents you have rated.

Build the judgment set first:

1. Pick 50 to 200 real queries from your search logs: the head (most frequent), a slice of the torso, and known problem queries. A rule of thumb: 50 is the minimum for a metric that does not swing on noise.
2. For each, rate the top results on a graded scale, for example 0 irrelevant, 1 related, 2 relevant, 3 perfect.
3. Store the judgments in version control next to the query template.

Then evaluate, ideally with a template so the queries under test are exactly the production query:

GET products/_rank_eval
{
  "templates": [
    {
      "id": "prod_query",
      "template": {
        "inline": {
          "query": {
            "multi_match": {
              "query": "{{q}}",
              "type": "cross_fields",
              "fields": ["title^3", "brand^2", "description"]
            }
          }
        }
      }
    }
  ],
  "requests": [
    {
      "id": "wireless_headphones",
      "template_id": "prod_query",
      "params": { "q": "wireless headphones" },
      "ratings": [
        { "_index": "products", "_id": "sku-48113", "rating": 3 },
        { "_index": "products", "_id": "sku-10022", "rating": 2 },
        { "_index": "products", "_id": "sku-77001", "rating": 0 }
      ]
    }
  ],
  "metric": {
    "dcg": { "k": 10, "normalize": true }
  }
}

The available metrics are precision, recall, mean_reciprocal_rank, dcg (with normalize: true it is nDCG) and expected_reciprocal_rank. Choose deliberately:

The response fields that matter:

{
  "metric_score": 0.712,
  "details": {
    "wireless_headphones": {
      "metric_score": 0.684,
      "unrated_docs": [ { "_index": "products", "_id": "sku-55190" } ],
      "hits": [ ]
    }
  },
  "failures": {}
}

metric_score at the top is the average across queries; compare it before and after a change. Per-query scores show which queries a change helped and which it hurt, which is the whole point. unrated_docs is your maintenance queue: once a change brings unrated documents into the top k, the metric is no longer measuring them, and you must rate them before trusting the score. For precision, ignore_unlabeled (default false) treats unrated documents as irrelevant; that pessimism is safer than the alternative.

Worked example of the decision: a change raises overall nDCG at 10 from 0.712 to 0.731, but two of the 120 queries drop from 0.9 to 0.4. Look at those two before shipping. They are often the head queries that most traffic hits, and a gain spread thinly over the tail can lose more user satisfaction than it wins.

Failure modes and anti-patterns

Replica inconsistency deserves a note. Deleted documents still count in term statistics until segments merge, and primaries and replicas merge independently, so two copies of a shard can return slightly different scores. If your users page through results and see duplicates or gaps, pass a consistent preference value (for example the session id) so a user's requests go to the same shard copies.

A relevance tuning runbook

When someone reports "search results are bad":

1. Get the exact query. The real request body, not a description. Capture it from the application log or slow log.
2. Reproduce and explain. Run it with explain: true for the top results and the expected result. Write down why the winner won in one sentence.
3. Classify the cause. Analysis (the expected doc does not match: check _analyze on both the query and the field), query structure (wrong multi_match type, missing phrase clause, wrong minimum_should_match), field weights, a business signal, or a missing synonym.
4. Add the query to the judgment set with ratings for the documents involved. A bug report becomes a test case.
5. Run rank_eval on the current production query to get the baseline score, per query.
6. Change one thing. One boost, one type change, one synonym. Run rank_eval again.
7. Read the per-query deltas, not just the average. Investigate every query that dropped.
8. Check latency. Run the candidate query with the profile API on representative queries. A script_score or a phrase clause on a big field can cost more than it gains. See search latency diagnosis.
9. Ship behind a template or a flag so you can roll back without a deploy.
10. Rate the new unrated_docs that the change surfaced, so the next measurement is honest.

Checklist for a new index:

• Separate index and search analyzers; synonyms only in the search analyzer.
• Multi-fields for different matching needs (for example title stemmed and title.exact unstemmed). See mapping design.
• Static signals as rank_feature fields; raw numeric copies if you need to sort or aggregate.
• A search template for the production query, so rank_eval tests exactly what users run.
• A judgment set in version control before launch.

Key takeaways

• BM25 scores are per shard, per query and not comparable across queries; read them with explain, never threshold them.
• k1 controls term-frequency saturation and b controls length normalization (defaults 1.2 and 0.75); change them per field only for a clear structural reason, and treat it as a reindex.
• The multi_match type decides how fields combine and usually matters more than boost values; cross_fields and combined_fields solve terms spread across fields.
• Prefer rank_feature for static signals; if you use function_score, guard against multiplying by zero.
• Synonyms belong at search time in a synonym_graph filter, managed so they reload without reindexing.
• Every change is measured with _rank_eval against a versioned judgment set, and per-query deltas matter more than the average.

How Sparced helps

Sparced does not judge relevance or tune queries; relevance is a property of your data and your users, and only a judgment set can measure it. What it does surface is the cluster underneath your relevance experiments: the shard layout of each index from its shard and allocation reads, which matters because per-shard term statistics behave differently on a small, heavily sharded index. It also reports search thread pool rejections, heap pressure and circuit breaker trips from the thread pool, node and breaker endpoints, so you can tell a slow or failing evaluation run caused by the cluster from one caused by the query. Each finding comes with the affected indices and nodes, a fix command with the names filled in, and a check to confirm it.