Frontier models solve 86% of a brand-new puzzle game — videogame evals are dead
gleech · x · 2026-11-17
A Paradigm 3 report by Zanichelli, Hambly and Leech tests GPT-6 Astra and GPT-6.1 Sol on Order of the Sinking Star, a game released after their training:
- From 460 sampled puzzles (of 920, timing-based ones excluded), the models solve 86% given 1M tokens per task (Sol 73%, Astra adds 13%). With cross-puzzle context they'd likely clear all turn-based content.
- Caveat: 13% of puzzles were public before the models shipped.
- Their hypothesis: labs now densely sample the space of games for post-training — games are the cheapest RL environments, mechanics recombine, and models can generate new ones — starting at scale mid-year. Similar scores across labs (Astra 63% on ARC-AGI-3, Gemini 4 Argon 50%, Claude Fable 5.2 81%) suggest shared environment vendors.
- Implication: eval pre-saturation is the norm in 2026; videogames can no longer measure generalisation.
Related event: GPT-6 models solve 86% of a brand-new puzzle game, study finds(2 posts)→
More from Research
- Probing activations restores diversity in distilled image and video generators — _akhaliq · 2026-11-17
- Perturbing inputs improves LM pretraining distillation, new paper shows — DimitrisPapail · 2026-11-17
- ReDiverse restores output diversity in few-step distilled diffusion models — anand_bhattad · 2026-11-17
- Inkling 2 Small hits 81.7% on ARC-AGI-2, beating DeepSeek V4.1 Flash but trailing on ARC-AGI-3 — teortaxesTex · 2026-11-17
- AI-formalized math spans Galois's 1832 last letter, fixing errors in a dozen published papers — stevenstrogatz · 2026-11-17
- FLUX 3 trains on images, video and sound: BFL on visual intelligence — Latent Space · 2026-11-17