Novel Claim or Déjà Vu? Rethinking "Contamination-Free'' Dynamic Evaluation for Multimodal Automated Fact-Checking
Haorui He, Xinwen Chen, Dacheng Wen, Reynold Cheng, Francis C. M. Lau, Yupeng Li
cs.CL, cs.AI, cs.MM
2026-07-26
Dynamic, post-cut-off benchmarks for fact-checking still contain 17–29% of claims models already know, inflating Macro-F1 by up to 11.34 points and flipping which model wins.
Multimodal automated fact-checking (MAFC) systems send large language models to retrieve evidence from the web and judge whether a claim is true. To score them you first need a set of claims as test items, and the items themselves are the problem. The dominant static benchmark, AVeriTeC, holds claims from before October 2020. Today's models already saw those events during pretraining, so they can answer from memory without ever going online, and scores get inflated. This is the familiar "data contamination" problem.
The recent fix is the "dynamic benchmark": collect only claims published after a model's knowledge cut-off, on the assumption they are unseen and therefore clean. This ACM MM 2026 paper challenges that assumption. Is it really true that a model knows nothing about events that happened after its cut-off? The answer is far less clean than assumed.
The authors build a new benchmark, ClaimReview2025Q4: 901 English-language claims published in Q4 2025 by IFCN-accredited fact-checking agencies, deduplicated and cleaned. On the static side they keep AVeriTeC's 491 claims, the latest from October 2020.
The test for contamination is direct. Contamination means a model substitutes its own parametric knowledge for retrieved evidence, so the authors have the model write a fact-checking article for the claim while fully offline, then measure how closely the evidence it produces from memory matches the evidence human checkers actually relied on. Three steps:
A high match means the model already carries enough evidence to resolve the claim, so the claim is not really "new" to it. The same pipeline doubles as a decontamination tool for existing benchmarks.
Six models are tested: GPT-5.2, GPT-4o-Mini, Gemini-3.0-Pro/Flash, DeepSeek-V3.2, and Qwen3.5-122B-A10B. The fact-checking system is fixed to DEFAME (ICML 2025), retrieving through Google web and image search, top 3 pages.
The one-line conclusion: dynamic benchmarks are cleaner than static ones, but not by enough.
The most striking number is the residual contamination. On ClaimReview2025Q4, even with only post-cut-off claims, 17.09%–29.30% are still judged potentially contaminated (intersection of six models and three metrics; Qwen3.5-122B-A10B is the worst). Going dynamic removes only 2.92–11.16 percentage points of contamination on average.
Why does a model know post-cut-off claims? Two routes. Some claims attach directly to pre-cut-off facts: the U.S. EMTALA medical law passed in 1986, the English town of Scarborough canceling its New Year fireworks in 2022, a 2020 TV-licence policy — "new" claims on old facts. Others are synthesizable: a claim about Barron Trump's 2028 Senate eligibility collapses into two public, pre-cut-off facts (senators must be at least 30, and he was born in 2006).
Inflation varies a lot across models. Splitting matched claims into contaminated and uncontaminated subsets (same DEFAME system):
| Model | Contaminated Macro-F1 | Uncontaminated Macro-F1 | Drop |
| Gemini-3.0-Flash | 61.22% | 49.88% | −11.34 |
| Qwen3.5-122B-A10B | 57.48% | 47.16% | −10.31 |
| Gemini-3.0-Pro | 61.07% | 52.50% | −8.57 |
| GPT-4o-Mini | 56.41% | 46.82% | −9.59 |
| GPT-5.2 | 57.68% | 52.81% | −4.87 |
| DeepSeek-V3.2 | 54.79% | 52.57% | −2.22 |
The asterisk marks a bootstrap test with p<0.05. GPT-5.2 and DeepSeek-V3.2 show no significant drop; the other four all do. Qwen3.5 has the largest accuracy decline, −10.88 points.
Contamination also flips the ranking. On the contaminated subset, Gemini-3.0-Flash leads Macro-F1 (61.22%); on the uncontaminated subset, the lead passes to GPT-5.2 (52.81%). The system you thought was strongest may simply be the one with the best memory.
There is a mechanism behind this. In Table 6, Qwen3.5's average "exploration" reformulations on uncontaminated claims is 1.24, against 0.89 on contaminated ones. Contaminated, the model skips the back-and-forth of re-querying and jumps to its memorized answer.
Finally the authors carve out 154 claims uncontaminated for all six models under all three metrics, a strictly controlled set. Every model lands below 56% Macro-F1; the best is DeepSeek-V3.2 at 55.93%. Fact-checking is nowhere near solved.
For anyone building benchmarks, this is a concrete warning: filtering by timestamp is not enough, and "dynamic equals clean" is a fragile assumption. Any fact-checking or retrieval-augmented benchmark built on the post-cut-off or LiveBench recipe may overstate capability by 5 to 11 Macro-F1 points and even reshuffle model rankings.
For fact-checking system builders, the picture is sobering. Once contamination is stripped out, the strongest model still sits around 56% Macro-F1, and nearly 70% of the data is the "Refuted" class. The task has substantial headroom.
The by-product is also practical. The offline-generate-then-match contamination pipeline is reusable as a decontamination tool for existing benchmarks, not just a means to this paper's conclusions.
The authors' own caveats: similarity matching catches only explicit evidence overlap and is likely conservative on subtler latent contamination; only English claims are tested; the knowledge cut-offs of the open-source models, DeepSeek and Qwen, are unknown, so contamination can only be judged from observable evidence. Thresholds were set on 180 claims (20% of the dynamic set), and evidence extraction defaults to GPT-4o-Mini, cross-checked with GPT-5-Nano with stable results.
One concern is worth flagging. The downstream comparison fixes DEFAME as the only system, and the contamination label itself depends on GPT-4o-Mini's evidence extraction, so two layers of LLM judgment are stacked without a clear accounting of how their errors compound. The 154-claim controlled set is also small and 68% Refuted, so the top-model call there (DeepSeek-V3.2) should not be read as settled.