How would you design an automated factuality checker for generated text at scale?

Define factual for the product. A synopsis can be wrong in two ways. It can state something false, or every sentence can be true while the overall picture misleads, for example by centring a side character or implying a romance is real when it starts as a fake arrangement. Netflix's post separates these as literal errors and implicature errors. Write both into the rubric, and collect a benchmark of items labelled pass or fail by expert annotators, with their rationales.

Baseline: one holistic judge. Netflix's previous pipeline gave a reasoning model all title metadata, text assets and a detailed rubric, ran it five times, and failed the synopsis if any run failed it. The post names two problems: cost, with five calls over contexts of 100k+ tokens, and misaligned reasoning, where the model failed synopses for nitpicks but talked itself out of real nuanced errors.

Cascade design from the post.

1. Specialized first-stage judges. A literal judge asks whether the synopsis is literally true. An implicature judge asks whether it gives a false picture to a new viewer. Netflix reports this worked better than repeating one generic judge and hoping sampling surfaces subtle errors; identical calls only reduce sampling noise, not shared reasoning mistakes.
2. Confidence gating. Each judge outputs a verdict, a rationale and a confidence. High-confidence verdicts are kept; low-confidence ones escalate, like extra airport screening for bags that trigger the alarm.
3. Router and atomic claims. The judges also list concessions: assumptions, doubts the evidence cannot resolve, and contradictions they dismissed. A router rewrites each as a self-contained atomic claim with the evidence needed, and sends it to a web judge for external facts or to a focused recheck against the source material.
4. Overturn rules. A FAIL becomes PASS only if every concession, including the original failure reasons, is supported. A PASS becomes FAIL if at least one concession is contradicted.

# Verified with Python 3 on 2026-11-09 (stub judges stand in for LLM calls).
def check(synopsis, literal_judge, implicature_judge, router, stage2):
    first = [literal_judge(synopsis), implicature_judge(synopsis)]
    passed = all(v.passed for v in first)          # any-fail aggregation
    if all(v.confidence == "high" for v in first):
        return passed                              # easy case: stop here
    claims = router([c for v in first for c in v.concessions])
    results = [stage2(claim) for claim in claims]  # web or source recheck
    if not passed and results and all(results):
        return True                                # FAIL -> PASS: every doubt resolved
    if passed and not all(results):
        return False                               # PASS -> FAIL: a concession contradicted
    return passed

Context matters. For episode synopses, Netflix added a full-show summary to the prompt. Without it, the post says the model tended to over-fail because it could not tell whether an episode detail fitted the wider storyline.

Human in the loop. Opinion: the gold set is the contract. Keep sampling production verdicts for annotator review, route disagreements and low-confidence cases to people when stakes are high, and feed corrected rationales back into the benchmark. Netflix lists using the rationale to correct faulty synopses, and multimodal inputs, as future work.

Alternate interview scenarios

"Your LLM judge runs five identical calls and fails the item if any run fails. Why might that still miss subtle errors?" Repeated calls only average out sampling noise. If the prompt leads every run down the same flawed reasoning, all five agree on the same wrong verdict, so you pay five times for one opinion. Self-consistency sampling helps when different reasoning paths reach the right answer, which is why Netflix saw runs disagree mostly on nuanced items. To catch systematic misses, diversify the judges instead: give each a specific error type to hunt, such as literal falsehood versus misleading implication, and recheck the specific doubts they raise. Keep any-fail aggregation if recall on bad items matters most, and measure what it costs in false fails.