DiG-bench: 70 Text Games Expose Reasoning Gaps in Frontier Models

Princeton, MIT, and other institutions have jointly released DiG-bench (Discovery Games Benchmark), a new benchmark specifically designed to test the scientific discovery and rule-exploration capabilities of large language models. Tests reveal that despite recent significant advancements, frontier models still perform poorly when faced with extremely simple text-based exploration problems, exposing shortcomings in core AI reasoning.

已确认

• 要点 Benchmark Design: DiG-bench includes 70 brand-new interactive text-based games requiring agents to discover unknown rules through experimentation to clear them. Because the environment is purely text-based, it eliminates extra interference like visual understanding, precisely testing the model's discovery capabilities in its strongest domain: natural language.
• 要点 Model Performance Gap: There is a significant gap between closed-source models (such as Opus 5 and Fable) and open-source models (such as Kimi K3), with closed-source models holding a substantial lead. However, even the best-performing models struggle with the hardest level 6 and 7 games.
• 要点 Framework Limitations: Test results indicate that utilizing existing Agent frameworks did not effectively enhance the models' scientific discovery capabilities.

为什么重要

• 要点 This benchmark directly probes AI's foundational discovery abilities in its native domain (natural language). It reveals that once stripped of visual and other interferences, current frontier LLMs still exhibit obvious flaws when tackling seemingly simple logic and rule-exploration tasks, offering a new perspective for evaluating true AI reasoning levels.

2026-08-12 ~ 2026-08-12 · 13 related posts

• Frontier Models Stumped by Simple Games: DiG-bench Tests AI Discovery — jcrwhittington · 2026-08-12
• DiG-bench: Evaluating AI Discovery via 70 Hidden-Rule Text Games — jcrwhittington · 2026-08-12
• Inside DiG-bench: How Text Games Evaluate AI Scientific Discovery — jcrwhittington · 2026-08-12
• Announcing DiG-bench: Evaluating AI Discovery in Pure Text Environments — jcrwhittington · 2026-08-12
• Frontier AI Models Struggle with Simple Discovery: DiG-bench Tests 70 Games — jcrwhittington · 2026-08-12
• [source] DiG-bench: Frontier AI Models Still Stumped by Simple Text-Based Discovery Games — jcrwhittington · 2026-08-12
• [source] DiG-bench Results: Closed-Source Models Lead by Leaps, Agent Frameworks Offer No Benefit — jcrwhittington · 2026-08-12
• [source] DiG-bench Releases 21 Browser Games, Challenging Devs to Fix AI Flaws — jcrwhittington · 2026-08-12
• DiG-bench: A Text-Based Benchmark for Evaluating Frontier LLM 'Discovery Capabilities' — AndrewLampinen · 2026-08-12

4 near-duplicate retellings: jcrwhittington · jcrwhittington · jcrwhittington · misovalko