Elasticsearch kNN in Production: HNSW, Quantization, Memory

Approximate kNN search in Elasticsearch is easy to demo and easy to get wrong at scale. The demo works because a few hundred thousand vectors fit in memory on a laptop. Production slows down or falls over for three predictable reasons: the vector data does not fit in the operating system page cache, too many small segments each carry their own HNSW graph, and recall was never measured, so nobody knows what num_candidates is buying. The answer is a sizing model you compute before you buy hardware, a quantization choice that matches your recall needs, a search configuration tuned against exact results, and hybrid ranking that combines vectors and BM25 instead of replacing one with the other. This article walks through each with the arithmetic shown.

The mental model: graphs per segment, memory outside the heap

A dense_vector field that is indexed for kNN is stored by Lucene as a Hierarchical Navigable Small World (HNSW) graph. Four facts shape everything else:

1. There is one graph per segment. A shard with 40 segments holds 40 separate graphs, and a kNN query searches each one and merges the results. Fewer, larger segments mean less work per query.
2. The graph is built at index time. Inserting a vector means searching the graph for its neighbors and linking it in, which is CPU-heavy. Merging segments rebuilds graphs, which is also CPU-heavy. Vector indexing is much slower than indexing plain documents of the same size.
3. Search wants the vectors and the graph in the page cache. HNSW search hops between nodes almost at random. If those reads hit disk, latency goes from milliseconds to something users notice. This memory is the operating system's file cache, not the JVM heap, which changes how you size nodes.
4. Every copy pays the full cost. With document replication, each replica indexes the documents itself and builds its own graphs. A replica doubles both the indexing CPU and the page cache footprint of the field.

A kNN search runs on each shard: it collects num_candidates candidates per shard by walking the graphs, keeps the best k, and the coordinating node merges the per-shard top-k lists into the global top-k.

Mapping a dense_vector field

PUT docs-v1
{
  "mappings": {
    "properties": {
      "title":     { "type": "text" },
      "tenant_id": { "type": "keyword" },
      "embedding": {
        "type": "dense_vector",
        "dims": 768,
        "index": true,
        "similarity": "cosine",
        "index_options": {
          "type": "int8_hnsw",
          "m": 16,
          "ef_construction": 100
        }
      }
    }
  }
}

The parameters that matter:

The default index_options.type for float vectors has changed across releases as quantization matured: quantized int8_hnsw became the default in 8.x, 9.1 moved vectors of 384 or more dimensions to bbq_hnsw, and later 9.x releases prefer the disk-based bbq_disk type where the license allows it. byte and bit vectors default to plain hnsw. Do not rely on the default: set it explicitly, and check what an existing index actually uses with:

GET docs-v1/_mapping/field/embedding?include_defaults=true

Neither similarity nor dims can be changed on an existing field. The quantization type can be changed in place only in one direction along a fixed chain (flat to int8_flat to int4_flat to bbq_flat to hnsw to int8_hnsw to int4_hnsw to bbq_hnsw, with m kept or increased), and existing segments keep their old format until they are rewritten by a merge or reindex, so plan most vector mapping changes as a reindex (see zero-downtime reindexing).

Memory sizing: the arithmetic

Elastic's documentation gives the off-heap memory that the vectors and graph need to stay in the page cache. Per vector field, per shard copy (the quantized rows use the per-vector correction bytes from the current documentation; older 8.x pages gave a smaller constant of 4 bytes for int8 and int4, which changes little):

Worked example

Assumptions: 20 million passages, 768-dimension embeddings, m = 16, one replica.

Graph: 20,000,000 * 4 * 16 = 1,280,000,000 bytes = 1.28 GB per copy.

Float vectors: 20,000,000 * 768 * 4 = 61,440,000,000 bytes = 61.44 GB per copy. Plus the graph: 62.72 GB. With one replica: 125.4 GB of page cache across the cluster.

int8: 20,000,000 * (768 + 16) = 15,680,000,000 bytes = 15.68 GB. Plus the graph: 16.96 GB per copy, 33.9 GB with a replica.

int4: 20,000,000 * (384 + 16) = 8.0 GB. Plus graph: 9.28 GB per copy, 18.6 GB with a replica.

bbq: 20,000,000 * (12 * 8 + 14) = 20,000,000 * 110 = 2.2 GB. Plus graph: 3.48 GB per copy, 7.0 GB with a replica.

Now the node side. Say three data nodes with 64 GB RAM each and a 30 GB heap (following the guidance to keep heap at no more than half of RAM and below the compressed object pointer threshold, which is near 32 GB and varies by JVM). Each node has at most about 34 GB outside the heap, less what the OS and other files need. Call it 25 GB of usable page cache per node, 75 GB across the cluster, as a stated assumption.

• Float: 125.4 GB needed, 75 GB available. Does not fit. Expect disk reads on most queries.
• int8: 33.9 GB needed. Fits with room for the rest of the index.
• bbq: 7.0 GB. Fits easily, and the cluster could be much smaller.

Two practical conclusions follow. First, on vector-heavy nodes a smaller heap often serves better, because the heap does not hold vectors and the page cache does; size the heap for your aggregations, request load and field data, not for the vectors. Second, the quantized formulas cover what search must read, not what is stored. Quantized indices still keep the original float vectors on disk (they are needed for rescoring and to requantize during merges). Disk usage for int8 in the example is roughly 61.44 + 15.68 + 1.28 = 78.4 GB per copy, before _source.

Do not forget _source

On 8.x and older 9.x indices the vector is also stored in _source as JSON text, which can be larger than the binary vector. Since 9.2, new indices default to index.mapping.exclude_source_vectors: true: vectors are kept out of the stored _source, left out of search and get responses, and rehydrated from the vector data (at float precision) when needed, for example by a reindex. The setting can only be chosen at index creation. Older guidance excluded vectors with _source.excludes. Be careful with that approach: an excluded vector is not returned by searches and cannot be recovered by a reindex from that index, so you would need to re-embed from the source system. Check what your version supports before choosing.

Quantization: what it costs and what it saves

Quantization is computed per segment from the data in it. For int8 and int4, the confidence_interval option controls how outliers are clipped when choosing the quantization range; leave it unset unless measurement says otherwise.

The standard way to recover recall lost to aggressive quantization is to gather more candidates with the cheap quantized vectors and rescore them with the full-precision ones. Recent releases expose this directly as a rescore_vector option with an oversample factor, either in the mapping's index_options or per query in the knn section (the query value wins). The factor multiplies k: the search gathers k * oversample candidates with the quantized vectors and rescores them against the raw floats. Values must be above 1.0 and below 10.0, or exactly 0 to disable it; on older versions you can achieve the same with a rescore clause using a script_score on the field. The cost is more raw-vector reads, which come from disk unless the floats also fit in cache.

Querying: k, num_candidates and filters

POST docs-v1/_search
{
  "knn": {
    "field": "embedding",
    "query_vector": [0.012, -0.044, 0.091],
    "k": 10,
    "num_candidates": 100,
    "filter": { "term": { "tenant_id": "acme" } }
  },
  "_source": ["title"],
  "size": 10
}

(The vector is abbreviated; it must have exactly dims values.)

• k is how many nearest neighbors to return overall.
• num_candidates is how many candidates each shard considers. It must be at least k and at most 10,000. In recent releases both are optional: k defaults to the request size, and num_candidates to 1.5 times k (capped at 10,000).
• filter is applied during the graph search, so you get k results that match the filter. This is different from a post_filter, which would filter after the top-k is chosen and can leave you with fewer than k hits.
• similarity (optional) sets a minimum similarity, useful when you would rather return nothing than a poor match.

You can also put a knn query inside the query DSL (for example in a bool with other clauses), available in recent 8.x releases. The top-level knn section is the most common form and is what the hybrid examples below use.

The _score you get back is a transformed similarity so that it is positive and higher is better. For cosine it is (1 + cosine) / 2; for l2_norm it is 1 / (1 + distance squared). Keep that in mind when you combine vector scores with BM25 scores: they live on different scales.

Choosing num_candidates

The trade-off is recall against latency, roughly linear in num_candidates. A rule of thumb is to start at 10 times k and measure. Higher ef_construction and m produce a better graph so that a smaller num_candidates reaches the same recall, which shifts cost from query time to index time.

Measure recall, do not guess it. Take 100 representative queries, compute the exact top-k with a brute-force script_score query, and compare with the approximate results:

POST docs-v1/_search
{
  "size": 10,
  "query": {
    "script_score": {
      "query": { "term": { "tenant_id": "acme" } },
      "script": {
        "source": "cosineSimilarity(params.qv, 'embedding') + 1.0",
        "params": { "qv": [0.012, -0.044, 0.091] }
      }
    }
  }
}

Recall at 10 is the size of the overlap between the exact and approximate top-10, divided by 10, averaged over the queries. Worked example: if over 100 queries the approximate results share on average 9.3 of the exact top 10, recall at 10 is 0.93. Then raise num_candidates from 100 to 200 and repeat. If recall moves to 0.96 and p95 latency moves from 18 ms to 26 ms (your numbers will differ), you have a concrete trade to decide on, not a guess.

The exact query is a full scan over the matching documents, so run the recall test off-peak or on a copy.

Filters and selectivity

A filtered kNN search walks the graph but only accepts matching documents. When the filter is very selective, the walk visits many non-matching nodes. Lucene handles the extreme case by switching to an exact scan of the matching documents in that segment, which is fast when few documents match. The awkward zone is a moderately selective filter on a large index, where neither the graph walk nor a scan is cheap. If one filter value (a tenant, a language) dominates your traffic, consider giving it its own index, so its graph contains only its own vectors. See multi-tenancy patterns.

Indexing and segments

Because graphs are per segment and graph building is expensive, ingest strategy matters more for vectors than for text.

• Bulk load with replicas at 0 and add them afterward. Each replica would otherwise build its own graphs during the load. Adding a replica later copies finished segments through peer recovery instead.
• Raise the refresh interval during bulk loads (for example "index.refresh_interval": "30s" or -1 during an initial load, restoring the default of 1s or your chosen value afterward). Fewer refreshes mean fewer tiny segments with tiny graphs to merge later.
• Force merge read-only indices. For an index that will not receive more writes, POST docs-v1/_forcemerge?max_num_segments=1 leaves one graph per shard and gives the best query latency. It is CPU-heavy and should run off-peak. Do not force merge an index still receiving writes. See indexing performance.
• Size shards with vectors in mind. A shard's vector memory and its merge cost both grow with vector count. The general shard sizing guidance still applies (see shard sizing), but for vector-heavy indices the page cache per node, not disk, is usually the binding constraint.

You can preload vector files into the page cache at shard open with the index.store.preload index setting, which takes a list of file extensions. Elastic's kNN tuning guidance is to preload only what must stay in RAM: for quantized HNSW, the quantized vectors (veq, or veb for BBQ) and the graph (vex), and not the raw vectors (vec). That avoids slow first queries after a restart but only helps when the data fits in memory in the first place; preloading too much can make search slower.

Hybrid search: BM25 and vectors together

Pure vector search is weak on exact identifiers, rare names, part numbers and anything the embedding model never saw. Pure BM25 is weak on paraphrase. Hybrid search runs both and combines them. There are two approaches.

Linear combination

Put a query and a knn in the same request. Elasticsearch takes the union of both result sets and sums the scores, each multiplied by its boost:

POST docs-v1/_search
{
  "query": {
    "match": { "title": { "query": "reset admin password", "boost": 0.3 } }
  },
  "knn": {
    "field": "embedding",
    "query_vector": [0.012, -0.044, 0.091],
    "k": 20,
    "num_candidates": 200,
    "boost": 0.7
  },
  "size": 10
}

The problem is scale. BM25 scores are unbounded and vary per query; the vector score for cosine is between 0 and 1. A fixed 0.3/0.7 weighting means different things on different queries. Linear combination can work if you normalize, but it needs tuning per query class.

Reciprocal rank fusion

RRF ignores scores and combines ranks:

rrf_score(doc) = sum over result sets of 1 / (rank_constant + rank_of_doc_in_that_set)

with rank_constant defaulting to 60. In recent releases you express it with retrievers:

POST docs-v1/_search
{
  "retriever": {
    "rrf": {
      "retrievers": [
        { "standard": { "query": { "match": { "title": "reset admin password" } } } },
        {
          "knn": {
            "field": "embedding",
            "query_vector": [0.012, -0.044, 0.091],
            "k": 50,
            "num_candidates": 200
          }
        }
      ],
      "rank_window_size": 50,
      "rank_constant": 60
    }
  },
  "size": 10
}

Worked example with rank_constant 60:

B wins because both systems like it, which is the property you want. A document found by only one system can still rank well if it is at the very top of that list. rank_window_size is how many results from each retriever take part; documents outside it contribute nothing, so set it larger than your page size.

Measure hybrid relevance with a judgment set and the rank_eval API, exactly as for text search. See relevance tuning.

Recent releases can also embed the query text inside Elasticsearch with a query_vector_builder that calls a deployed text embedding model, and offer a semantic_text field type that handles chunking and embedding on ingest. Both move inference load onto your cluster, so size ML nodes for it.

Failure modes

The model-mismatch case deserves emphasis. Embeddings from two different models live in unrelated spaces. Changing the model is always a full re-embed and reindex into a new index, followed by an alias swap.

Production runbook and checklist

Before launch:

1. Fix the embedding model and record its name and version on each document.
2. Pin dims, similarity and index_options (type, m, ef_construction) explicitly in the mapping or template.
3. Compute page cache needs with the formulas above for every copy, and compare with RAM minus heap on the data nodes that will hold the shards.
4. Bulk load with replicas 0 and a long refresh interval, then restore both.
5. Force merge if the index is read-only.
6. Measure recall at k against exact script_score results on at least 100 queries; record the num_candidates that meets your target.
7. Build the hybrid query (RRF to start) and measure relevance with a judgment set.

When kNN latency degrades:

1. Check segment counts per shard. Many small segments point to ingest settings or a missed force merge.
2. Check disk read activity on the data nodes during search. Sustained reads mean the working set does not fit.
3. Recompute the memory formula with today's vector count. Collections grow; page cache does not.
4. Check whether replicas or new indices moved onto nodes already near their memory limit.
5. Check num_candidates in the application. A change from 100 to 1,000 is a 10x cost increase that often slips in unnoticed.
6. Confirm the heap is not oversized on vector nodes, starving the page cache.

Key takeaways

• HNSW graphs are per segment and live in the OS page cache, not the heap; every replica pays the full memory and indexing cost.
• Size before you buy: num_vectors * dims * 4 for float, num_vectors * (dims + 16) for int8, plus num_vectors * 4 * m for the graph, per copy.
• Quantized indices still keep the float vectors on disk; the savings are in what search must keep in memory.
• Pin index_options explicitly; defaults have changed between releases.
• Tune num_candidates against measured recall from exact script_score results, not by feel.
• Use knn.filter, not post_filter, to get k filtered results.
• Start hybrid search with RRF; it needs no score normalization.

How Sparced helps

Sparced does not measure recall or tune kNN parameters. What it reads in its single pass is the state that decides whether vector search can be fast: node heap usage and the configured heap against RAM on each node, shard placement per node from the shard and allocation endpoints, and search thread pool queues and rejections. A node past 85% heap is flagged with the evidence, which helps separate a genuine heap problem from a vector memory problem, since vectors live off-heap in the page cache. Each finding names the indices, shards and nodes involved, gives a fix command with those names filled in, and a check to confirm the fix. It writes nothing to the cluster.