REKEY: New Benchmark Exposes VLM Score Inflation from Memorization
jiqizhixin · x · 2026-08-14
AI benchmarks often suffer from data contamination, where test data leaks into training, allowing models to memorize answers rather than truly understand. To address this, researchers introduced REKEY.
REKEY is a live benchmark that randomly swaps crucial details in real images at test time, forcing Vision-Language Models (VLMs) to rely on actual perception. Across 8 top VLMs, scores dropped by 9.5 to 18.8 points compared to standard tests, demonstrating the extent of memorization-induced score inflation.
More from Research
- Terence Tao Uses AI to Prove Sendov's Conjecture, Lean Formalization Completed — stevenstrogatz · 2026-08-14
- ARC Benchmark: Frontier Models Absorbing Harness Patterns for Better Reasoning — mhmazur · 2026-08-14
- Embodied AI Blind Spot: 9 VLMs Know How to Repair but Fail to Proactively Spot Issues — rohanpaul_ai · 2026-08-14
- Anthropic Researcher: Lean Kernel to Be Bulletproof by AI Within a Year — geoffreyirving · 2026-08-14
- Kevin Murphy's Solo Paper: Using LLMs and Bayesian Methods for Efficient Scientific Discovery — burny_tech · 2026-08-14
- Compute with Human Brain Cells: Parasma CEO Proposes Radical Energy-Efficient Alternative to Silicon — bosmeny · 2026-08-14