The 30 second answer
I want golden signals for the search API and for Elasticsearch itself: QPS, p95/p99 latency, error rate, and saturation of CPU, heap, and thread pools. I break those down per client and per query class, especially aggregations. Product metrics like zero-result rate and click success catch relevance regressions. Index lag and bulk reject rates catch write-path pain. Traces must show fan-out across query builder, base search, and re-rankers.
RED / golden signals for the edge API. Rate, errors, duration (p50/p95/p99), plus saturation. DDIA's latency discussion is the reason to alert on percentiles, not averages.
Elasticsearch-specific. Query and fetch latency; search / search_coordination thread-pool rejections; JVM heap and GC; young vs old GC; refresh and merge times; bulk indexing latency and rejects; circuit breakers. High-cardinality aggregations deserve their own dashboard.
Per-client attribution. Zalando's incident follow-ups: per-client slow logs via X-Opaque-Id, dashboards for slow and aggregating queries, rate limits by client type. Internal tools are often the real DDoS.
Relevance & product health. Zero-result rate, exact-match rate, add-to-cart after search, revenue proxies, and A/B guardrail metrics. Attribute funnel events so a KPI drop can be traced to a view/query id (MICES monitoring discussion).
Data path health. Consumer lag on the event bus, Flink checkpoint duration/state size, dead-letter depth, and "time from product write to searchable document."
Opinion: page the on-call on user-facing SLO burn and on thread-pool rejections; ticket relevance metric drifts for daytime analysis unless they breach a hard business guardrail.
Sources:
- Elasticsearch — Monitor a cluster
- Elasticsearch — X-Opaque-Id (opaque id / headers conventions)
- Zalando Engineering — The Day Our Own Queries DoS'ed Us
- Google SRE — Monitoring distributed systems