Why can a few facet requests per second overload a search cluster that serves thousands?

Zalando's engineering blog published a post-mortem of a search outage caused by its own traffic. It is a good basis for an interview discussion because it combines Elasticsearch internals with incident response.

What the post describes. On a Sunday, one Elasticsearch cluster serving two of the largest markets became slow: queries that took milliseconds took seconds, filters showed zero results, and CPU spiked. There were no deployments, no jump in client traffic and no write spike, and other clusters were healthy, which the post credits to grouping markets on separate clusters. The usual playbook (longer cache expiry, disabling non-critical requests, lower query termination thresholds, scaling coordinator and data nodes) did not help. The cause, found later through traces, was an internal application that, because of a maintenance workload and a bug, sent about 50 times its usual query volume. That was still only 20–100 requests per second against a cluster used to thousands, and the queries were facets aggregating on SKU, a unique product id.

Why so few requests hurt so much.

1. One bucket per value. A terms aggregation builds a bucket for every distinct value it sees. On a field like SKU the number of distinct values approaches the number of matching documents. The docs warn that larger size values use more memory and push the request towards the search.max_buckets limit.
2. Over-fetching per shard. For accuracy, each shard returns its top shard_size terms, which defaults to size * 1.5 + 10, and the coordinating node must hold and merge all of those lists. The docs note this costs bytes over the wire and memory on the coordinator.
3. Global ordinals. Terms aggregations on keyword fields use global ordinals, a per-shard mapping that lives in heap; the docs warn that aggregations on high-cardinality fields "can use a lot of memory" and can trip the field data circuit breaker.
4. Thread-pool starvation. Per-shard query and aggregation work runs on the search thread pool. When heavy aggregations occupy its threads and fill its queue, Elasticsearch rejects other search work, so ordinary product queries fail too. The post says high-cardinality terms aggregations did not benefit from the newer parallel collectors and ran as heavy work on the search pool.

Why monitoring missed it. The post lists the reasons: the queries were syntactically valid, came from a known internal service, were low in volume, and were untagged, so they were indistinguishable from legitimate facet traffic. Per-client volume monitoring existed, but the volume was "peanuts".

Guardrails worth naming.

• search.max_buckets limits the buckets in a single response; the documented default is 65,536, and requests over the limit fail. The post does not say what value the cluster used, so do not assume the default was in effect.
• The request circuit breaker stops a single request whose structures would exceed a heap share, documented as 60% of JVM heap by default, but Elastic says breakers "do not track all memory usage" and give only incomplete protection. The post describes CPU saturation and full queues, not a breaker trip.
• Application-side limits: which clients may facet, on which fields, with what size. Facets on identifiers have no business meaning, which the post points out.
• For "all distinct values", Elastic recommends the composite aggregation, which pages through buckets instead of building them all at once.

What the team changed, according to the post. Rate limiting and isolation by client type, client identifiers in slow logs through the X-Opaque-Id header, per-client dashboards for slow queries, aggregating queries and aggregation sizes, application-side query limits with dynamic thresholds, and runbooks that separate read load from write load and cover blocking a misbehaving client and applying search.max_buckets cluster-wide.

The load-shedding ladder for an overloaded search cluster is answered in How do you keep search available when queries or updates overload the cluster?

Alternate interview scenarios