The 30 second answer
I check intrinsic health first: a histogram of similarities on held-out data quickly shows a collapsed space where everything looks similar. Then I measure where the embedding is used. For retrieval that is recall at k on held-out, time-split orders, confirmed by an online test of the product metric. For a feature, I run an ablation: train the downstream model with and without it on the same data.
Intrinsic checks. Analogy tests and benchmark suites such as MTEB grade the space itself. MTEB covers many tasks and found no single text embedding method dominates all of them, so a leaderboard rank does not prove fit for your task. A cheaper sanity check catches collapse: when training satisfies positives by pulling everything together, similarities bunch near one and retrieval becomes noise. Weak or too few negatives are one common cause. A histogram of pairwise similarities on held-out data shows it immediately.
Extrinsic checks. Measure the embedding where it is used, matched to the use.
- Retrieval: recall at k on held-out orders from after the training window: of the products shoppers actually added, how many appear in the top k? Then an online test on the feature's metric, such as upsell conversion.
- Feature: the question is incrementality. Train the downstream model with and without the embedding and compare on the same held-out data.
- RAG: retrieval recall separately from answer quality.