Product search system design 12 questions in 4 chapters

I separate discovery from commerce writes. Search must be fast, highly available, multilingual, and good enough that empty results are rare, while price and stock stay eventually consistent. Booking-style strong consistency belongs to checkout, not the search index. The hard trade-offs are freshness versus indexing cost, precision versus recall, and personalization versus a strict latency budget. I would state SLOs for p99 latency, index lag, and relevance quality up front.

Product creates and updates stay in an operational store with clear ownership. Search gets its own denormalized documents built asynchronously from change events. That is CQRS in practice: the write side protects invariants, the read side optimizes for filters, facets, and ranking features. I accept eventual consistency and design explicit lag SLOs plus backfill. I would not let checkout read stock from Elasticsearch.

I prefer change events over polling. CDC from the database or a transactional outbox gives at-least-once delivery into a bus, then a stream job joins product, price, and stock into one search document. Thin events that only carry an id force a re-fetch; fat events risk fan-out of huge payloads. Idempotent consumers, poison-event handling, and a scheduled backfill are mandatory. I would ask which fields must be near real time before sizing the pipeline.

I shard by document id inside each index and usually isolate markets or market groups so one region's load cannot take others down. Writes go through aliases so reindex and blue-green upgrades are atomic. Primaries are sized for growth; replicas buy read capacity and availability. Analyzers are per language. I call out over-sharding and high-cardinality aggregations as self-inflicted outage risks.

First I clean and understand the query: tokenize, spell-correct, detect entities like brand or category, and decide redirects. Then base search retrieves a candidate set with lexical matching, filters, and optionally vectors. A later stage re-ranks with business rules and models, and may blend promotions. Facets are a separate aggregation-shaped call. Every optional enrichment sits behind a latency budget with a simpler fallback.

Retrieval decides which products are allowed into the candidate set using inverted indexes, filters, and maybe vectors. Ranking orders that smaller set with heavier features and models. Separating them keeps the expensive stage bounded and lets lexical and semantic recall improve without rewriting business rankers. I retrieve thousands of candidates cheaply, then re-rank a few hundred. Debugging also splits cleanly: missed item versus wrong order.

I never ship on vibe. Offline, I use labeled query–product judgments and metrics like NDCG or recall at a cutoff as a regression suite. Online, I A/B test engagement and revenue while watching for position-bias and UI layout confounds. LLM-as-judge can scale labels if calibrated against humans. I also track zero-result rate and complaint themes. A ranking win that hurts diversity or novelty may still be wrong for the business.

I cache aggressively on the edges that repeat: popular query strings, common filter combinations, and rendered facet payloads. Coordinator-side result caches help aggregations. I keep user-specific personalized scores lightly cached or computed late so one shopper does not poison another. Price and stock badges need short TTLs or live re-checks. Cache is a performance layer, not the source of truth, so every entry needs TTL and a bust path on ingest.

I treat search as a layered system with a control plane above Elasticsearch. First I shed load at the edge: fewer parallel fan-outs, more caching, and skip heavy ML or promotion calls. I rate-limit by client and block expensive aggregations. For write storms I throttle ingest and protect refresh. Isolation by market cluster limits blast radius. Runbooks must separate read overload from write overload because the fixes differ.

I scale reads by replicas and by splitting markets onto separate indices or clusters closer to shoppers. I keep primary shard counts intentional so I can add nodes without heroic resharding. Cross-region active-active search usually means independent indexes fed by replicated events, accepting divergence. Global one-index designs hurt latency and blast radius. Capacity plans should include Cyber Week style peaks for both queries and catalog updates.

I want golden signals for the search API and for Elasticsearch itself: QPS, p95/p99 latency, error rate, and saturation of CPU, heap, and thread pools. I break those down per client and per query class, especially aggregations. Product metrics like zero-result rate and click success catch relevance regressions. Index lag and bulk reject rates catch write-path pain. Traces must show fan-out across query builder, base search, and re-rankers.

Search is a derived view, so it lags the write side by design. I set an explicit lag SLO for price and stock, push those fields on a faster path than rare attribute edits, and always re-validate at checkout. Users may see a brief mismatch; the buy path must not. After a merchant edit I may read the primary service for that id. I never pretend the index is strongly consistent.