The 30 second answer
I never ship on vibe. Offline, I use labeled query–product judgments and metrics like NDCG or recall at a cutoff as a regression suite. Online, I A/B test engagement and revenue while watching for position-bias and UI layout confounds. LLM-as-judge can scale labels if calibrated against humans. I also track zero-result rate and complaint themes. A ranking win that hurts diversity or novelty may still be wrong for the business.
Relevance work without measurement is just rearranging bias.
Offline evaluation. Build a judgment list: for sampled queries, human (or calibrated model) labels of which products are relevant. Relevant Search Ch9–10 describes judgment lists, test-driven relevance, and monitoring clickthrough after deploy. Classic IR metrics: precision/recall, and ranking metrics such as NDCG@k. Zalando's ranking paper reports offline search gains such as Recall@500 and NDCG@500 versus a prior GBT candidate generator — useful as an example of reporting at a retrieval cutoff that matches the ranker input size.
Online evaluation. A/B tests on engagement, conversion, revenue, novelty, and diversity. The same paper reports online browse and search lifts and notes when revenue changes were not significant — cite carefully, do not round into marketing claims. Watch for position bias and UI changes (items per row) that shift clicks and poison training labels; Zalando's MICES talk calls this out in monitoring discussions.
LLM-as-judge. Zalando announced an LLM-as-judge relevance assessment approach (Nov 2024) comparing GPT-class judges on tens of thousands of examples, aiming at cheaper multilingual labeling than pure human labor. Opinion: treat LLM judges as a scalable labeling aid that still needs human calibration and clear guidelines — not as ground truth.
Operational quality metrics. Zero-result rate, reformulation rate, add-to-cart after search, p99 latency, and error budgets. A relevance change that blows the latency SLO is a regression even if NDCG rises.
Sources:
- Relevant Search — measuring relevance (Ch9–10)
- arXiv 2409.02856 — offline/online results
- Zalando Engineering — LLM-as-a-judge relevance assessment announcement
- Microsoft — Experimentation / A/B testing guidance (general methodology reference)