2026-08-05
FActScore decomposes long-form LM output into atomic facts and checks each against a knowledge source. ChatGPT biographies are only 58% supported, collapsing to 16% for rare entities.
When a language model writes a long passage (a biography, a summary, a report), its factual errors are rarely whole-passage right or wrong. A single sentence often mixes a correct claim with a fabricated one, which breaks the usual evaluation toolkit. A sentence-level true/false label forces a binary call on something that is half-true, and paying humans to check every claim is slow and expensive. FActScore starts from the premise that to measure factuality you first have to cut the text fine enough.
Two moves. First, a generation is broken into atomic facts, each carrying one checkable piece of information ("Bridget Moynahan is an actress", "she trained at the American Academy of Dramatic Arts"). A single sentence often yields several; ChatGPT averages 4.4 atomic facts per sentence. At this granularity, half-true statements can be scored separately instead of lumped together.
Second, each atomic fact is checked against a reliable knowledge source (English Wikipedia in the paper) and marked supported or not. FActScore is the share of atomic facts that are supported. It measures precision only, not recall: a model that refuses to answer or writes very little can score high, which is not necessarily desirable.
To avoid manual checking, the paper also trains an automated estimator: it retrieves relevant passages from the knowledge source and uses a language model to judge each atomic fact. The estimator approximates human FActScore with an error rate under 2%.
On biography generation, three then-commercial models were all error-prone:
| Model | Search | FActScore |
| InstructGPT | No | 42.5% |
| ChatGPT | No | 58.3% |
| PerplexityAI | Yes | 71.5% |
Search-augmented PerplexityAI scored highest, but even with direct access to Wikipedia it reached only 71.5%, because it often copied incorrect search results. The most striking finding is the rarity effect: on common entities ChatGPT's facts are about 80% supported, but on rare entities it collapses to 16%. On the long tail the model is largely inventing.
The automated estimator was then run over 6,500 generations from 13 models, work that would have cost roughly $26K in human annotation. One finding: among 7B-scale open models, Alpaca and Vicuna (around 40%) clearly beat MPT-Chat (30%) and StableLM (17%), so the training recipe matters more than raw size for factuality.
FActScore turns "is this model making things up" from a vague impression into a reproducible, fine-grained number that can even tell you which category of facts fails. It became the de facto standard for factual-precision evaluation, and many later model papers report it. From 2026's vantage point, as agent systems start to emit long reports and research summaries in bulk, the atomic-fact-by-atomic-fact approach is still the most direct framework for asking whether their output can be trusted; the only change is that the knowledge source swaps from Wikipedia to a more specialized retrieval corpus.
Honestly, FActScore is an evaluation tool; it does not by itself make models better. But it gave factuality a comparable yardstick for the first time, and much of the later hallucination-reduction work rests on the ability to measure.
The authors are explicit: all experiments live in one domain, people biographies against Wikipedia, because biographies are objective and Wikipedia covers them well. Whether the assumptions hold for current events (knowledge source: a news collection) or scientific claims (a paper corpus) is not fully validated. It measures precision, not recall, so overly abstinent models are overrated. And the support judgment depends on the knowledge source's quality and currency; when Wikipedia is wrong or silent, the estimator is wrong too.
The estimator's bias also varies across the model being evaluated; on search-augmented PerplexityAI, whose outputs are nondeterministic, it is notably harder to score, so the estimator is not fully model-agnostic and may need recalibration on a new model set.