How do you train a two-tower retrieval model with in-batch and hard negatives?

Towers. Instead of lookup rows, each side is a network that turns features into a vector. The cart tower reads the items in the cart and maybe recent history; the product tower reads title, category, brand and price. Both output the same dimension, and the loss only sees their dot product, so that score means "how likely this product completes this cart". Because towers compute vectors, unseen carts and new products get embeddings.

Positives. Take completed orders and hold one item out. The rest of the cart is the query and the held-out item is the positive.

In-batch negatives and InfoNCE. Score every cart in a batch against every product. The diagonal holds true pairs; off-diagonal products are free negatives. InfoNCE treats each row as a classification: a softmax over the row and the negative log-probability of the diagonal item, which is ordinary cross-entropy. Larger batches give more negatives. In-batch sampling over-represents popular items as negatives; Yi et al. correct the batch loss for this sampling bias, using item frequencies estimated from streaming data.

# Verified with Python 3 and NumPy 2.5 on 2026-11-09.
import numpy as np

def info_nce(cart_vecs, item_vecs, temperature=0.05):
    # Row i of cart_vecs pairs with row i of item_vecs; other rows are in-batch negatives.
    q = cart_vecs / np.linalg.norm(cart_vecs, axis=1, keepdims=True)
    k = item_vecs / np.linalg.norm(item_vecs, axis=1, keepdims=True)
    logits = q @ k.T / temperature               # B x B cosine scores
    logits -= logits.max(axis=1, keepdims=True)  # numerical stability
    log_probs = logits - np.log(np.exp(logits).sum(axis=1, keepdims=True))
    return -np.mean(np.diag(log_probs))          # cross-entropy, target = diagonal

Correctly paired carts give a loss near zero; shuffling the pairs makes it large.

Hard negatives. Random negatives soon become easy and give almost no gradient, so training loss flattens while retrieval stays weak. Hard negatives, items the current model ranks highly but the shopper did not buy, force fine distinctions. Finding them needs a search with the current model, so teams periodically snapshot it, re-embed the catalog and mine neighbours from that index; ANCE formalizes this with an asynchronously refreshed ANN index. The very top neighbours are often unlabeled true positives, so skip or down-weight them, as RocketQA's denoised hard negatives do.

A complete answer. Say what a positive is, where negatives come from, and that evaluation uses orders placed after the training window.

Alternate interview scenarios

"Why use InfoNCE rather than triplet loss?" Triplet loss takes an anchor, a positive and a negative and asks the positive to be closer than the negative by at least a margin. Once a triplet satisfies the margin its loss and gradient are zero, so training effort moves to triplets that are still wrong. That is intuitive, but it needs careful triplet mining; FaceNet used online mining for that reason. InfoNCE compares each positive against every other item in the batch at once, so with large batches it gets many negatives for free. I would pick InfoNCE for large-batch retrieval training and keep triplet loss as a reasonable alternative when batches are small or data naturally comes as triplets.

"How would you build reverse image search without click labels?" Make positives by augmentation. Crop an image slightly, recompress it, shift colours or rotate it, and treat each version as a positive for the original, with other images in the batch as negatives. The model learns to ignore those distortions, so a blurry screenshot lands next to the catalog photo. When both sides are the same kind of input, one shared encoder serves both sides, which is a Siamese setup, the structure Sentence-BERT uses for text.