DiG-bench: 70 Text Games Expose Frontier AI's Discovery Gaps

Princeton, MIT and other institutions have released DiG-bench (Discovery Games Benchmark), a set of 70 handcrafted text-only interactive games for evaluating LLMs' scientific discovery and rule-exploration abilities. Tests show frontier models have improved markedly in recent months yet still fail on some extremely simple text exploration problems; closed-source models lead clearly, and existing agent frameworks brought no gains. 21 games are now playable in browsers with an open API.

Confirmed

• Benchmark design: DiG-bench contains 70 distinct text-only game worlds with hidden rules and win conditions, requiring agents to discover rules through experimentation with minimal hints. All games are handcrafted and previously unpublished online (21 public, 49 held for safety evaluation), spanning 7 difficulty levels; some are minimal string worlds where the player "walks" as arrows.
• Model performance: Frontier models improved significantly over recent months but still fail on some very simple text exploration problems. Closed-source models such as Opus 5 and Fable lead clearly yet struggle on the hardest levels 6–7; a clear gap separates closed-source Opus 5 from open-source Kimi K3.
• Framework limits: Existing agent frameworks did not effectively improve models' scientific discovery ability.
• Open resources: The 21 public games are playable in the browser, with an API allowing developers to test any model; the team invites the community to build frameworks that solve all games or narrow the gap between small open-source models and SOTA.
• Discussion: AndrewLampinen joined the post-release discussion, responding to claims that each game was first solved by at least one human.

Why it matters

• The researchers' premise is that for AI to make genuine scientific discoveries rather than merely combine existing tools, it must first prove exploration ability in the simplest settings. The text-only environment removes visual confounds and directly probes discovery in the language domain where LLMs are strongest, revealing clear deficits on seemingly simple rule-exploration tasks and offering a new lens on AI's true reasoning level.

2026-08-12 ~ 2026-08-13 · 14 related posts

• Frontier Models Stumped by Simple Games: DiG-bench Tests AI Discovery — jcrwhittington · 2026-08-12
• DiG-bench: Evaluating AI Discovery via 70 Hidden-Rule Text Games — jcrwhittington · 2026-08-12
• Inside DiG-bench: How Text Games Evaluate AI Scientific Discovery — jcrwhittington · 2026-08-12
• Announcing DiG-bench: Evaluating AI Discovery in Pure Text Environments — jcrwhittington · 2026-08-12
• Frontier AI Models Struggle with Simple Discovery: DiG-bench Tests 70 Games — jcrwhittington · 2026-08-12
• [source] DiG-bench: Frontier AI Models Still Stumped by Simple Text-Based Discovery Games — jcrwhittington · 2026-08-12
• [source] DiG-bench Results: Closed-Source Models Lead by Leaps, Agent Frameworks Offer No Benefit — jcrwhittington · 2026-08-12
• [source] DiG-bench Releases 21 Browser Games, Challenging Devs to Fix AI Flaws — jcrwhittington · 2026-08-12
• DiG-bench: A Text-Based Benchmark for Evaluating Frontier LLM 'Discovery Capabilities' — AndrewLampinen · 2026-08-12
• New DiG-bench Tests AI Discovery: Frontier Models Still Stumped by Simple Problems — AccBalanced · 2026-08-13

4 near-duplicate retellings: jcrwhittington · jcrwhittington · jcrwhittington · misovalko