The 30 second answer
Instead of one verdict for a whole text, I break it into short self-contained claims and check each against a trusted source, so one wrong detail is not hidden by many correct ones. FActScore scores the share of supported atomic facts; SAFE checks each fact with search; NLI models test whether the source entails a sentence. Decomposition still misses misleading-but-true framing, so I keep a holistic implication check.
Why decompose. Long text mixes supported and unsupported statements, so a single binary verdict is coarse and hard to explain. Breaking it into atomic facts makes each check small and the rationale specific.
Techniques.
- FActScore splits a generation into atomic facts and reports the percentage supported by a reliable knowledge source, with an automated estimator that uses retrieval and a strong language model.
- SAFE uses a language model to split a long response into individual facts, sends search queries for each, and decides whether the results support it. It also proposes an F1-style metric that balances supported facts against a target response length.
- NLI-based consistency. Natural language inference classifies whether a premise entails, contradicts or is neutral to a hypothesis. SummaC showed that NLI works for summary consistency when the document and summary are split into sentences and pair scores are aggregated, because NLI models are trained at sentence level.
- SelfCheckGPT needs no external source: sample several answers from the model, and treat facts that are not consistent across samples as likely hallucinations. It suits open-ended generation, not checking against a known source.
Targeted decomposition. Netflix decomposes only where the first-stage judges were unsure: the router turns their concessions into atomic claims, each with a field describing the evidence needed, and routes it to a web judge or a source recheck. That keeps the cost of decomposition on the hard cases.
Limit. Atomic checks verify literal statements. A synopsis where every claim is true but the emphasis misleads passes every claim check, so a separate whole-text implicature check is still needed.
Sources (5)
- Min et al., FActScore: Fine-grained Atomic Evaluation of Factual Precision
- Wei et al., Long-form factuality in large language models (SAFE)
- Laban et al., SummaC: NLI-based inconsistency detection in summarization
- Manakul et al., SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection
- Netflix TechBlog: Fact-Checking at Scale (router and atomic-claim rechecks)