Is this Citation on Point?
Apurv Verma
ICML 2026
cs.DL, cs.CL
2026-08-13
Bloomberg-built benchmark corrupts real legal citations by swapping the pinpoint page within the same case; GPT-5.4 with high reasoning still misses 40% of these on court opinions, because models treat topical overlap as support.
When two attorneys were sanctioned in Mata v. Avianca (2023) for filing ChatGPT-fabricated citations, the legal AI field responded with benchmarks for hallucinated authorities. But fabricated cases are the easy failure: a database lookup catches them. The harder failure, largely untested until now, is a citation to a real case at a real page that does not support the proposition it is offered for. This is exactly what human cite-checking spends its hours on. The paper (ICML 2026, Apurv Verma of Bloomberg) builds a proposition-level citation support benchmark: given a legal proposition, its citation, and the cited page, decide whether the page actually supports the claim.
Two public corpora: CLERC (US federal court opinions, 2,000 citations) and BriefMe (legal briefs, 750). Citations are classified by function; only substantive ones (cited for a legal rule, 70%) are evaluated. The core design is controlled perturbation of real citations at three difficulty levels:
Hard mirrors real cite-checking: spotting a swapped authority is easy, verifying page-level support is not. Human validation shows 84-88% agreement with heuristic labels. The model receives the cited page text and outputs on point / not on point. Fourteen configurations were tested: GPT-4.1, GPT-4o, GPT-5, GPT-5.4 (standard and high reasoning), GPT minis, Claude Sonnet 4 / 4.6, Claude Opus 4.6, and Gemini 2.5 Flash / 2.5 Pro / 3.1 Pro. Metrics: recall on corrupted citations, false positive rate (FPR) on valid ones.
| Setting | Easy recall | Hard recall | FPR |
| GPT-5.4 (reasoning), CLERC opinions | 99.8% | 59.8% | 13.1% |
| GPT-5.4 (reasoning), BriefMe briefs | 99.4% | 82.0% | 14.4% |
| GPT-4o, CLERC (high-recall end) | 99.8% | 60.6% | 34.9% |
| Claude Opus 4.6, CLERC (low-FPR end) | 98%+ | 36.5% | 4.1% |
| Gemini 2.5 Pro, BriefMe | 99.1% | 82.7% | 16.3% |
Cross-cutting findings:
Manual coding of failure rationales shows three patterns: in about two-thirds of misses, the model invents support, claiming the cited page "expressly states" text that is not there; models treat topical overlap as sufficient; and quoted text is not checked, with 92% of GPT-5 false negatives containing quotes absent from the cited page.
A page-grounded prompt (verbatim quote check, page-level verification, topic-vs-support distinction) raises Hard recall for every model: GPT-5.4 reasoning goes 59.8% to 73.2% on CLERC (FPR 13.1% to 19.5%), GPT-5 goes 77.0% to 91.0% on BriefMe (FPR 14.4% to 22.5%). Every gain in recall buys a rise in false positives across all models: added skepticism targets all citations, not only corrupted ones. Models can be made more skeptical, not selectively skeptical.
For legal AI builders, this is a direct self-audit template: verifying that a cited case exists is nowhere near enough, and the best current configuration (GPT-5.4 high reasoning plus page-grounded prompting) still leaves 27% misses and 20% false alarms on opinions, far from replacing human cite-checking. For the broader LLM community, the finding transfers cleanly: recognizing the right topic and verifying support for a specific proposition are distinct capabilities, and current models conflate them. That distinction applies to RAG and search grounding generally, where topical match is not support. The evaluation recipe itself, controlled perturbation of real citations graded by difficulty with paired recall/FPR reporting, is reusable in other domains: medical guidelines, financial disclosures, academic citation checking.
Author-acknowledged: coverage limited to substantive citations in US federal opinions and briefs; support is treated as binary, while practitioners also weigh binding versus persuasive authority, factual analogy, and subsequent history; corruptions may remain on point by chance, so recall may be understated; relational misuse (distinguishing or criticizing authority) is untested; no end-to-end comparison against human review workflows or costs.
Two more caveats on close reading. With 84-88% human agreement, the benchmark carries 12-16% label noise, so a few points of model ranking on CLERC sits inside the noise band. And the "prompting is not enough" conclusion rests on a single hand-written page-grounded prompt; whether retrieval augmentation or a trained verifier can break the recall-FPR tradeoff is untested.