The 30 second answer
I spend expensive calls only where they change the answer. Easy items get one cheap pass and stop when the judge is confident; uncertain items escalate to more or stronger checks, which is an LLM cascade. I verify that the confidence signal is calibrated on the gold set, measure cost, calls per item and median latency per tier, and accept slower hard cases if throughput allows.
Where cost comes from. Calls per item, context length and model size. Netflix's baseline paid for five long-context reasoning calls on every synopsis, though easy items got near-identical verdicts on every run.
Cascades. FrugalGPT describes LLM cascades as one of three cost strategies, alongside prompt adaptation and LLM approximation: try cheaper steps first and stop when the answer is reliable enough. Netflix's version keeps two specialized first-stage judges for every item and escalates only low-confidence items to a router and a smaller Stage-2 model.
Confidence must be checked. A gate is only as good as its confidence signal. Tian et al. found verbalized confidence from RLHF models often better calibrated than token probabilities, while Xiong et al. found verbalized confidence tends to be overconfident. Measure, on the gold set, how often high-confidence verdicts are wrong before trusting the gate.
Netflix's reported results. On show-level synopses, total cost fell from 1573.46 to 748.34, which the post states as a 52.4% reduction, and calls per item from 5 to 3.38. On episode-level synopses the cascade cost 259.60 versus 306.90 for the holistic pipeline, which the post states as 15% cheaper, with 2.93 calls per item. The post also reports P50 latency rising from 17.3 to 33.2 on show-level and 17.2 to 45.1 on episode-level, which Netflix considered reasonable because latency was not the main target; the table does not state the unit.
Opinion. For offline content checks, throughput and cost matter more than median latency. For an inline user-facing check, set a latency budget per tier and fall back to holding content for review when the budget runs out.