Medium system-design machine-learning

How does matrix factorization learn item embeddings, and why start with it?

The 30 second answer

Put past orders in a cart-by-product matrix and learn two thin matrices whose row dot products reconstruct it. Purchases pull a cart and its products together, while empty cells push weakly, because not buying is weak evidence. The product rows become item vectors. It is cheap and well understood, so it is a strong baseline that proves later complexity is worth it.

Setup. Rows are carts or users, columns are products, and a cell is 1 when the cart held the product. Most cells are empty, and an empty cell is not a true zero: the shopper may simply not have found the item. Learn U (one short row per cart) and V (one short row per product) so that U·Vᵀ approximates the matrix. Because each row is small, the model cannot memorize every order and must find shared structure.

Loss. Observed purchases should score high; unobserved cells should score low but with a smaller weight. That is the implicit-feedback formulation of Hu, Koren and Volinsky: a binary preference plus a confidence weight that grows with the evidence. Alternating least squares fixes V and solves U exactly, then the reverse; stochastic gradient descent also works. After training, oral-care products cluster in one area and coffee items in another, learned only from carts.

Why mention it in an interview. It is a cheap, explainable baseline. It tells you whether a two-tower model or graph model is worth its training and serving cost.

Limits. It is a lookup table. A cart built ten minutes ago and a product listed this morning have no row until the next training run. Averaging the cart's product vectors helps but blurs which combination is present. That motivates models that compute vectors from features.

Sources (1)
  1. Hu, Koren and Volinsky, Collaborative Filtering for Implicit Feedback Datasets (ICDM 2008)