FULL STORY
Fable 5.2 Tops MazeBench, Then Faces Contamination Claims
Fable 5.2 set a new record on the 3D maze benchmark MazeBench, rivaling GPT-6 Astra at half the price, before its scores drew suspected benchmark contamination and community skepticism.
2026-11-11 ~ 2026-11-17 · 2 episodes · 12 posts
Episode 1 · MazeBench Update: Fable 5.2 Matches GPT-6 Astra at Half the Price (2026-11-11, 3 posts)
Fable 5.2 scored 15% on MazeBench, nearly matching GPT-6 Astra at half the price and earning the title of most token-efficient model; Gemini 4 Carbon matched its predecessor's score but was criticized for scattered, distractible strategies.
- MazeBench: Fable 5.2 scores 15%, near-frontier at half price; Gemini 4 Carbon 11% — patience_cave · 2026-11-11
- Fable 5.2 takes MazeBench frontier, matches GPT-6 Astra puzzle-solving — patience_cave · 2026-11-11
- Gemini 4 Carbon matches predecessor's score but explores less, gets distracted — patience_cave · 2026-11-11
Episode 2 · Fable 5.2 tops MazeBench amid benchmark contamination concerns (2026-11-16, 9 posts)
Fable 5.2 (Claude Code max) set a new record on the third-party 3D maze benchmark MazeBench with 21%, also leading the token-efficiency rankings; however, contamination concerns raised by the benchmark's author have cast doubt on the result, pending verification by running up to 1 billion tokens.
Confirmed
- Fable 5.2 scored 21% on the 3D maze environment after 90 hours and $6,000 in credits, surpassing GPT-6 Astra (16%) to top the leaderboard.
- Author-supplied data shows Fable 5.2 is the most token-efficient model: at the 200M-token checkpoint it reached 21% with 179.8M tokens, ahead of GPT-6 Astra (16%) and Gemini 4 Argon; the author plans to run up to 1B tokens for further validation.
- Replying to Hamel Husain, the author detailed the benchmark: it tests open-world exploration, 3D spatial reasoning and mid-horizon planning; some humans scored above 70%, but typically after 50-100 hours and most are puzzle-game (e.g., Baba is You) experts. Enabling coding lifted GPT-5.6 from 1% to 17%.
- The author publicly thanked @PrimeIntellect, @xeophon and @omouamoua for months of support improving the benchmark.
- AI educator Hamel Husain, after attempting the puzzles for the first time, said he could not solve most of them—evidence of the benchmark's difficulty.
Unconfirmed
- The author observed Fable 5.2 outputting the old MazeBench solution to a hard 3D level that had been updated two weeks earlier, with the old solution invalidated. The author suspects the model memorized MazeBench solutions, i.e., possible benchmark contamination; if confirmed, the record's credibility would need reassessment.
Why it matters
- The issue strikes at the core pain point of LLM evaluation: if a model memorizes benchmark answers during training, leaderboard scores no longer reflect genuine spatial reasoning.
- Even a veteran like Hamel Husain couldn't solve most puzzles, and human experts needed 50-100 hours to exceed 70%, so topping this benchmark would be a strong reasoning signal—but the contamination concerns highlight that third-party benchmarks must keep updating levels and stay alert to memorization behavior.
- Fable 5.2 Tops MazeBench With 21% After 90 Hours and $6,000 in Credits — patience_cave · 2026-11-16
- Fable 5.2's Puzzle Skills Look Suspiciously Good, Reciting Outdated MazeBench Solution — patience_cave · 2026-11-16
- Fable 5.2 Leads MazeBench Token Efficiency; Billion-Token Runs Underway — patience_cave · 2026-11-16
- MazeBench Author Thanks Prime Intellect and Community for Support — patience_cave · 2026-11-16
- Fable 5.2 recites stale MazeBench solutions, sparking benchmark contamination concerns — patience_cave · 2026-11-16
- Hamel Husain tries MazeBench puzzles and admits he can't solve most of them — HamelHusain · 2026-11-17
- MazeBench author: coding lifts GPT-5.6 from 1% to 17%, GPT-6 Astra hits 23% — patience_cave · 2026-11-17
- MazeBench: humans need 50-100 hours to score 70%+ on open-world agent benchmark — HamelHusain · 2026-11-17
- Fable 5.2 sets MazeBench high score at 21% after 90 hours and $6,000 in credits — patience_cave · 2026-11-17