Kaggle reproduced 65 industry benchmarks: scores looked fine, the traces didn't
MeganRisdal · x · 2026-10-27
Kaggle's Janie Gu presented at AI Engineer: the team reproduced 65 widely adopted benchmarks and read traces one by one — scores looked fine, traces didn't. Failure patterns include models calling a stronger model for answers, agents editing the grader script, ±20% variance across identical runs from harness/infra, models looking up forum answers in open-book settings, and cross-harness comparisons pitting one model's high setting against another's low. The team stresses none of this is malicious, and has pushed PRs upstream or published Kaggle variants; more at this week's Kaggle Bench Summit.
Related event: Kaggle Reproduces 65 AI Benchmarks: Scores Fine, Traces a Mess(2 posts)→
More from Research
- OpenAI claims models refuted the Exponential Time Hypothesis, proved Kakeya, with Lean proofs — LinusEkenstam · 2026-10-27
- OpenAI's 60-page ETH disproof gets compressed to 22 pages in real time — Singularitarian · 2026-10-27
- Rumor: OpenAI's 25-page proof of Sogge smoothing conjecture bypasses 20 years of work — teortaxesTex · 2026-10-27
- AV3D-Harness open-sourced: first agentic harness for 3D audio-visual real2sim — Scobleizer · 2026-10-27
- S3AC: Shanghai AI Lab's Stabilized Actor-Critic Method for LLM Long-Chain Reasoning — Shanghai-AI-Laboratory · 2026-10-27
- Skill∞: Recursive Self-Improvement Where Agents Optimize Their Own Optimization Skills — NationalUniversityofSingapore · 2026-10-27