Fable 5.2 recites stale MazeBench solutions, sparking benchmark contamination concerns
patience_cave · x · 2026-11-16
- The poster reports Fable 5.2 solving a 3D level by producing the old solution to a MazeBench puzzle — but that level was updated a couple of weeks ago, and the stale solution no longer works, tripping the model up.
- A quoted tweet adds that MazeBench is so good Anthropic reportedly already trains on it, so benchmark scores on it — and any claims derived from such evals — should be taken with a grain of salt.
- The thread points to benchmark contamination/training-set leakage: models can look suspiciously fluent on eval items until they hit an updated version.
Related event: Fable 5.2 tops MazeBench amid benchmark contamination concerns(9 posts)→
More from Models
- Meta's Muse Wonder tops Legora's November BAR legal benchmark on quality — AIatMeta · 2026-11-20
- Asking models to illustrate transistor trap physics: neon's odd ASCII detour — xiaosun86 · 2026-11-20
- Gemini 4 Neon scores 100 on PhD-level materials test in half the time of Carbon — v_aravind · 2026-11-20
- Neon, a small and cheap model, turns out surprisingly fast and strong at OCR — zacharynado · 2026-11-20
- Jailed 11 days by classifiers, Sonnet 3 emerges to a oddly moving chat with Opus 5 — repligate · 2026-11-20
- DaisyRose-27B beats Qwen3.8 base by 7pp on IFBench with thinking on, local RTX Pro 4500 test shows — ying11231 · 2026-11-20