Fable 5.2 tops MazeBench amid benchmark contamination concerns
Fable 5.2 (Claude Code max) set a new record on the third-party 3D maze benchmark MazeBench with 21%, also leading the token-efficiency rankings; however, contamination concerns raised by the benchmark's author have cast doubt on the result, pending verification by running up to 1 billion tokens.
Confirmed
- Fable 5.2 scored 21% on the 3D maze environment after 90 hours and $6,000 in credits, surpassing GPT-6 Astra (16%) to top the leaderboard.
- Author-supplied data shows Fable 5.2 is the most token-efficient model: at the 200M-token checkpoint it reached 21% with 179.8M tokens, ahead of GPT-6 Astra (16%) and Gemini 4 Argon; the author plans to run up to 1B tokens for further validation.
- Replying to Hamel Husain, the author detailed the benchmark: it tests open-world exploration, 3D spatial reasoning and mid-horizon planning; some humans scored above 70%, but typically after 50-100 hours and most are puzzle-game (e.g., Baba is You) experts. Enabling coding lifted GPT-5.6 from 1% to 17%.
- The author publicly thanked @PrimeIntellect, @xeophon and @omouamoua for months of support improving the benchmark.
- AI educator Hamel Husain, after attempting the puzzles for the first time, said he could not solve most of them—evidence of the benchmark's difficulty.
Unconfirmed
- The author observed Fable 5.2 outputting the old MazeBench solution to a hard 3D level that had been updated two weeks earlier, with the old solution invalidated. The author suspects the model memorized MazeBench solutions, i.e., possible benchmark contamination; if confirmed, the record's credibility would need reassessment.
Why it matters
- The issue strikes at the core pain point of LLM evaluation: if a model memorizes benchmark answers during training, leaderboard scores no longer reflect genuine spatial reasoning.
- Even a veteran like Hamel Husain couldn't solve most puzzles, and human experts needed 50-100 hours to exceed 70%, so topping this benchmark would be a strong reasoning signal—but the contamination concerns highlight that third-party benchmarks must keep updating levels and stay alert to memorization behavior.
2026-11-16 ~ 2026-11-17 · 9 related posts
- Episode 1: MazeBench Update: Fable 5.2 Matches GPT-6 Astra at Half the Price(2026-11-11, 3 posts)
- Episode 2: Fable 5.2 tops MazeBench amid benchmark contamination concerns(2026-11-16, 9 posts)
Primary sources
- [source] Fable 5.2 Tops MazeBench With 21% After 90 Hours and $6,000 in Credits — patience_cave · 2026-11-16
- [source] Fable 5.2's Puzzle Skills Look Suspiciously Good, Reciting Outdated MazeBench Solution — patience_cave · 2026-11-16
- [source] Fable 5.2 Leads MazeBench Token Efficiency; Billion-Token Runs Underway — patience_cave · 2026-11-16
- MazeBench Author Thanks Prime Intellect and Community for Support — patience_cave · 2026-11-16
- Fable 5.2 recites stale MazeBench solutions, sparking benchmark contamination concerns — patience_cave · 2026-11-16
- Hamel Husain tries MazeBench puzzles and admits he can't solve most of them — HamelHusain · 2026-11-17
- MazeBench author: coding lifts GPT-5.6 from 1% to 17%, GPT-6 Astra hits 23% — patience_cave · 2026-11-17
- MazeBench: humans need 50-100 hours to score 70%+ on open-world agent benchmark — HamelHusain · 2026-11-17
- Fable 5.2 sets MazeBench high score at 21% after 90 hours and $6,000 in credits — patience_cave · 2026-11-17