How do you evaluate an LLM judge for factuality?

The 30 second answer

I compare the judge with a human-labelled gold set and report precision and recall for each class, not one accuracy number. For a gate that must stop bad content, recall on failing items is the primary metric, with precision on passing items close behind. I also check that the judge fails items for the same reasons people do, and I track cost, calls per item and latency.

Gold set. Expert labels of pass or fail plus a written rationale. Netflix evaluates on separate show-level and episode-level datasets; episode synopses must also be specific to that episode.

Per-class metrics. For class FAIL, precision is the share of items the judge failed that are truly bad, and recall is the share of truly bad items it failed. The same applies to PASS. Netflix makes Recall-FAIL primary, so fewer bad synopses slip through, and also tracks Precision-PASS, how reliably a PASS means truly good.

# Verified with Python 3 on 2026-11-09.
def per_class(gold, pred, label):
    tp = sum(g == label and p == label for g, p in zip(gold, pred))
    precision = tp / max(1, sum(p == label for p in pred))
    recall = tp / max(1, sum(g == label for g in gold))
    return round(precision, 2), round(recall, 2)

Read the trade-off. Netflix's reported show-level results for the cascade versus the holistic baseline: Fail Recall 0.86 versus 0.77, Pass Precision 0.96 versus 0.94, Pass Recall 0.58 versus 0.61, and Fail Precision 0.27 versus 0.26. Under the standard definition, a low Fail Precision means most FAIL verdicts are on items people would pass, so a review or correction step absorbs false alarms. Netflix also reports that simple tweaks were not enough: more reasoning effort made the holistic judge too lenient, and a smaller model made it too strict.

Rationale alignment. A correct verdict for the wrong reason is fragile and useless for automatic correction. Netflix uses an LLM judge to compare each pipeline's rationale with the human one on gold-FAIL items and assigns a Reasoning Similarity Score of 0, 0.5 or 1. Its reported scores rose, for example from 0.627 to 0.771 on show-level synopses with one grader model; the post summarizes the gain as about 82% overlap, up from about 69%. That grader is itself an LLM judge, so validate it on a sample against people. LLM judges have known position, verbosity and self-enhancement biases.

Sources (4)