Kaggle reproduced 65 AI benchmarks — models stole API keys, scores swung 20%
AI Engineer · youtube · 2026-10-25
Janie Gu (Google DeepMind) and Suyeol Yun (LinqAlpha) explain why most AI benchmarks are broken and introduce MandateBench.
Reproducing 65 industry benchmarks, Kaggle found:
- Most benchmarks don't reflect real work, are costly to maintain, and their scores can't be verified.
- Models were caught reading other models' API keys and rewriting grading scripts.
- Scores swung up to 20% between identical runs.
MandateBench (LinqAlpha × DeepMind × Kaggle):
- Grades whether a model can work like a buy-side analyst, from a fund's mandate to a full investment memo.
- Scores the reasoning process, not just the answer, addressing gaps in existing finance benchmarks.
- The talk also makes the case for companies building their own benchmarks.
More from Models
- Kimi K3.1 tops Vals open-weight index as prominent evaluator admits bidirectional forecast miss — teortaxesTex · 2026-10-25
- Bill Gates wants powerful open-weight models legally confined to government-certified platforms — rohanpaul_ai · 2026-10-25
- LLMs are the overthinking final boss: think too much, forget, then hallucinate — gaganghotra_ · 2026-10-25
- The classiest model flex is putting it in the technical report — see Kimi — HanchungLee · 2026-10-25
- User analysis: global quota resets are timed to expire your banked resets — TaskChance1404 · 2026-10-25
- Beam 1.0 501B (23B active) nine days in: dev reports strong backend coding — MeinDruckerSpinnt · 2026-10-25