The 30 second answer
I treat search as a layered system with a control plane above Elasticsearch. First I shed load at the edge: fewer parallel fan-outs, more caching, and skip heavy ML or promotion calls. I rate-limit by client and block expensive aggregations. For write storms I throttle ingest and protect refresh. Isolation by market cluster limits blast radius. Runbooks must separate read overload from write overload because the fixes differ.
Failure modes.
- Read amplification. One user request becomes many ES queries (facets, A/B variants, promotions). A buggy internal client can look "low QPS" yet destroy CPU with high-cardinality aggregations — Zalando's Dec 2025 post: ~20–100 RPS of bad traffic versus thousands of healthy RPS, rooted via distributed traces.
- Write / merge pressure. Bulk ingest + low
refresh_interval+ heavy merges raise heap and IO; search thread pools starve. - Tail latency under fan-out. DDIA emphasizes percentiles over means: scatter/gather makes p99 dominate user experience.
Mitigations to put on the whiteboard
- Admission control: rate limits per client type; reject or simplify expensive aggs (
search.max_buckets, application-side query limits). - Load shedding ladder: disable non-critical fan-outs → sample fewer requests into heavy rankers → fall back to lexical score only.
- Pool isolation: dedicated coordinating nodes; separate clusters for market groups.
- Observability hooks:
X-Opaque-Id(or equivalent) for per-client slow logs — called out in Zalando's follow-ups. - Ingest throttle: pause or slow bulk indexing during rescue; reduce replicas temporarily only with a clear plan to restore redundancy.
Opinion: "just add nodes" is last, not first, when the query shape is pathological. Fix the query taxonomy and client budgets first.
Sources:
- Zalando Engineering — The Day Our Own Queries DoS'ed Us
- Elasticsearch — Thread pools
- [Elasticsearch — Search settings (
max_buckets, etc.)](https://www.elastic.co/docs/reference/elasticsearch/configuration-reference/search-settings) - DDIA — percentile latency / tail latency (study map theme 6)
Sources (0)
See inline citations in the answer above.