Guides
Elasticsearch, explained the way it behaves in production
30 guides for the people who run the cluster: the model underneath, the arithmetic, the failure modes and the exact commands. Written for Elasticsearch 8.x and 9.x.
Architecture
Topology, tiers, replication and the shape of a cluster that survives.
Elasticsearch Capacity Planning for Logging: A Worked ExampleTurn GB per day of logs into node counts per tier: indexed size, replicas, retention, watermark headroom, shard counts and ingest rate, with all the math.Elasticsearch Node Roles and Cluster Topology for ProductionDesign node roles, master quorum, voting-only tiebreakers and zone awareness so an Elasticsearch cluster survives node and zone loss, with configs and sizing.Elasticsearch Cross-Cluster Replication and Search ArchitectureDesign remote cluster connections, run cross-cluster search with predictable latency, set up CCR followers, and fail over with a tested runbook.Elasticsearch Data Tiers: Hot, Warm, Cold, Frozen with ILMDesign hot, warm, cold and frozen tiers with ILM and searchable snapshots: tier preference, partial vs full mounts, a storage cost model and when frozen pays.Elasticsearch Multi-Tenancy: Index per Tenant vs Shared IndexPick an Elasticsearch multi-tenancy model with the shard math shown: index per tenant, shared indices with routing and filtered aliases, DLS, noisy neighbors.Elasticsearch Shard Sizing: A Capacity Model for Shard CountsCompute shard size and count for time-series and search indices, read per-node shard limits, and fix oversharding with the exact APIs and the arithmetic shown.Logging Architecture with Kafka and Elasticsearch: A Sizing GuideDesign a Kafka-buffered logging pipeline into Elasticsearch: partitions to consumers, backpressure, duplicates and the exactly-once myth, with sizing math.
Operations
Incidents, upgrades, snapshots and the work of keeping it running.
Elasticsearch Disk Watermarks, Flood Stage and Headroom PlanningHow the low, high and flood stage disk watermarks work, why indices go read_only_allow_delete, how max_headroom changes the math, and how much disk to plan.Elasticsearch ILM Troubleshooting: Fixing Stuck PoliciesRead _ilm/explain like a state machine, fix rollover alias and allocation errors, retry or move a stuck step, and change a policy on indices already in flight.Elasticsearch JVM Heap, G1GC and Circuit Breakers ExplainedSize the Elasticsearch heap correctly, read G1GC and _nodes/stats/breaker output, and trace a circuit_breaking_exception back to the request that caused it.Elasticsearch Red Cluster Runbook: Lost Primaries and RecoveryTriage order for a red Elasticsearch cluster: find lost primaries, then choose between waiting, allocate_stale_primary, allocate_empty_primary or a restore.Elasticsearch Rolling Upgrades: 7.17 to 8.x and 8.x to 9.xPlan and run an Elasticsearch rolling upgrade: deprecation checks, node order, allocation settings, client compatibility and why rollback means a snapshot.Elasticsearch Snapshots, SLM and Disaster Recovery PlanningHow incremental snapshots work, SLM schedules and retention, RPO and RTO arithmetic you can defend, restore strategies, and a monthly restore test.Fix Unassigned Shards in Elasticsearch with Allocation ExplainRead the allocation explain API decider by decider: same_shard, disk_threshold, filter, awareness, max_retry, node left. Find the cause and fix each one safely.Zero-Downtime Reindexing in Elasticsearch: Aliases and CutoverChange Elasticsearch mappings with no downtime: alias-first design, sliced and throttled reindex, catching up live writes, atomic cutover and a rollback plan.Monitoring Elasticsearch in Production: Metrics, Alerts, PagingSet up Stack Monitoring on a separate cluster, learn the metrics that predict incidents, set alert thresholds, and decide what pages versus what opens a ticket.
Performance
Heap, threads, indexing and search latency, measured and fixed.
Elasticsearch Indexing Performance: Bulk, Refresh and MergesRaise Elasticsearch indexing throughput with measured bulk sizes, refresh_interval, replicas, translog and auto IDs, and read the stats that show the limit.Elasticsearch Thread Pools and Rejections: What 429 Really MeansRead _cat/thread_pool correctly, find the real cause of write and search rejections, and build client backpressure instead of raising queue sizes.
Data modeling
Mappings, templates, data streams and changing them without downtime.
Elasticsearch Index Templates, Component Templates, Data StreamsHow composable templates merge, how priority resolves conflicts, how data stream backing indices roll over, and when a plain index is the better choice.Elasticsearch Mapping Design for Production: Types and LimitsDesign Elasticsearch mappings that stay fast and small: text vs keyword, doc_values, index: false, dynamic templates, strict mode, flattened and field limits.Cutting Elasticsearch Storage Costs: logsdb, TSDS and CodecsMeasure where Elasticsearch disk goes per field, then cut it with logsdb, TSDS, synthetic _source, best_compression, mapping trims, force merge and frozen.
Ingest
Logstash, Elastic Agent, ingest pipelines and buffered pipelines.
Elastic Agent and Fleet vs Beats: Architecture and MigrationChoose between Fleet-managed Agent, standalone Agent and Beats, size Fleet Server, design policies, and migrate from Beats without gaps or duplicates.Ingest Pipelines vs Logstash: Where to Parse Your LogsDecide where parsing belongs, ingest pipelines or Logstash, with grok vs dissect costs, on_failure patterns, the simulate API and ingest node sizing math.Logstash Pipeline Architecture: Workers, Queues and IsolationSize Logstash workers and batches, choose memory or persistent queues, use the dead letter queue and isolate outputs with pipeline-to-pipeline patterns.
Search
Relevance, aggregations and vector search in production.
Elasticsearch Aggregations at Scale: Accuracy, Memory, LimitsGet correct terms counts, page through every bucket with composite, size cardinality memory, stay under search.max_buckets and pre-aggregate with transforms.Elasticsearch Relevance Tuning: BM25, Boosts and rank_evalTune Elasticsearch relevance with evidence: read BM25 scores with the explain API, choose multi_match types, add business signals and measure with rank_eval.Diagnosing Elasticsearch Search Latency: Profile API to CachesFind where an Elasticsearch search spends its time: query vs fetch, slow logs, the profile API, the query and request caches, and the patterns that slow it.Elasticsearch kNN in Production: HNSW, Quantization, MemoryRun Elasticsearch vector search in production: size HNSW memory step by step, pick m, ef_construction and num_candidates, quantize safely and blend with RRF.
Security
TLS, roles, API keys and the least privilege that actually works.
Kibana and observability
Kibana at scale and monitoring the stack itself.