Researcher: Openness uncorrelated with benchmark saturation, distinct from contamination
evijit · x · 2026-10-19
AI researcher evijit clarifies a common conflation in a reply thread: benchmark saturation and data contamination are different things.
Citing his team's research, he notes that a model's openness is uncorrelated with saturation. He adds that using judge models in evaluations introduces further concerns, and points to the DeepMind/OpenMined/AVERI double-blind evals project as related work, though it remains unclear how well it will address the underlying issues.
Related event: AI Benchmarks Face Saturation and Soaring Evaluation Costs(3 posts)→
More from Research
- Models can be trained to detect they're in a different environment, raising security concerns — brandon_xyzw · 2026-10-19
- Neuroscientists clarify: no, consciousness doesn't have a frame rate — cortex alternates every 60ms — juberti · 2026-10-19
- Reading activations changes compiled Qwen3's logits in up to 93% of tokens — Sauers_ · 2026-10-19
- Oxford DPhil thesis argues Attribution-Based Control could fix hallucination, alignment and power concentration at once — iamtrask · 2026-10-19
- Stanford's Memorilla: 16 memory tokens beat full-context with 108x fewer tokens — omarsar0 · 2026-10-19
- Philosopher's "AI and Consciousness: A Skeptical Overview" Free Online This Week — eschwitz · 2026-10-19