MazeBench author: coding lifts GPT-5.6 from 1% to 17%, GPT-6 Astra hits 23%
patience_cave · x · 2026-11-17
The creator of MazeBench, an open-world maze benchmark, answered Hamel Husain's questions with concrete data:
- What it tests: open-world exploration, 3D spatial reasoning, and medium-horizon planning. Humans scoring above 70% typically needed 50-100 hours and deep sokoban-game experience (e.g. Baba is You).
- Tools matter: enabling coding boosts scores dramatically — GPT-5.6 Sol scored 17% with coding enabled vs 1% without. GPT-6 Astra was far less tool-dependent (23% vs 16%), hinting stronger models rely less on writing solvers.
- Design: built with the author's brother; each room encodes one tool-based mechanic, e.g. using a long box to reach far away and push another box into place.
Related event: Fable 5.2 tops MazeBench amid benchmark contamination concerns(9 posts)→
More from Models
- Meta's Muse Wonder tops Legora's November BAR legal benchmark on quality — AIatMeta · 2026-11-20
- Asking models to illustrate transistor trap physics: neon's odd ASCII detour — xiaosun86 · 2026-11-20
- Gemini 4 Neon scores 100 on PhD-level materials test in half the time of Carbon — v_aravind · 2026-11-20
- Neon, a small and cheap model, turns out surprisingly fast and strong at OCR — zacharynado · 2026-11-20
- Jailed 11 days by classifiers, Sonnet 3 emerges to a oddly moving chat with Opus 5 — repligate · 2026-11-20
- DaisyRose-27B beats Qwen3.8 base by 7pp on IFBench with thinking on, local RTX Pro 4500 test shows — ying11231 · 2026-11-20