Fable 5.2 reproduces outdated MazeBench solutions, sparking contamination suspicion
patience_cave · x · 2026-11-16
Duplicate post of the same event: Fable 5.2 reproduced the solution to an old version of MazeBench that was updated weeks ago, failing only where the old fix no longer works—though it could sometimes infer recent changes. The community asks whether Anthropic trained on MazeBench, a suspected benchmark-contamination case.
More from Models
- OpenAI brings Apollo, METR, Transluce, Goodfire in for embedded safety assessments — mark_k · 2026-11-17
- Vibe Check: Gemini 4 Carbon Codes Better Than Argon but Trails Fable 5.2 on Judgment — every · 2026-11-17
- New MKA Benchmark: 200 Expert-Level Multimodal Tasks Across 83 Occupations — osanseviero · 2026-11-17
- OpenAI Brings in Apollo, METR, Transluce and Goodfire for Embedded Safety Assessments — OpenAI · 2026-11-17
- MKA Benchmark: Gemini 4 Carbon Scores 54.3% at $5.63 Per Task vs Claude's $35.28 — _philschmid · 2026-11-17
- Gemini Decisions API Judges 1,000 Questions Over 256K Tokens With Zero Output Token Billing — _philschmid · 2026-11-17