Dynamic Benchmarks Still Leak: Up to 30% of Test Items Remain Contaminated

青稞AI · wechat · 2026-08-04

A research team from Hong Kong Baptist University and other institutions reveals that dynamic evaluation, which uses questions published after a model's knowledge cutoff date, still fails to fully eliminate data contamination.

Hidden Contamination Mechanisms

Old Knowledge in New Claims: Some newly published claims discuss old facts, or their answers can be easily deduced by combining existing knowledge the model already possesses.
High Contamination Rates: In dynamic benchmarks for multimodal automated fact-checking, 17.09%–29.30% of samples still show potential contamination risks.

Distorted Evaluation Results

Inflated Scores: Contaminated samples significantly boost model performance, artificially inflating Macro-F1 by up to 11.34 percentage points.
Incorrect Leaderboards: Contamination alters the relative ranking of models. The top-performing model changes when tested on a strictly uncontaminated subset.
Search Trajectory Shortcuts: When facing contaminated claims, models skip broad exploration and issue highly targeted queries, acting as if they already have the clues.

True Performance Post-Decontamination

On a rigorously filtered, contamination-free test set, the Macro-F1 of all current mainstream multimodal fact-checking systems falls below 56%, indicating that their ability to genuinely investigate unknown events is still limited.

Original post →

Dynamic Benchmarks Still Leak: Up to 30% of Test Items Remain Contaminated

More from Research