Frontier AI Models Struggle with Simple Discovery: DiG-bench Tests 70 Games
jcrwhittington · x · 2026-08-12
Princeton, MIT, and other institutions introduce DiG-bench, a novel benchmark of 70 text-based interactive games designed to evaluate the scientific discovery capabilities of AI agents.
- Test Design: Each game requires the agent to discover unknown rules through experimentation. Being purely text-based eliminates visual confounds, probing discovery capabilities directly in the natural domain of language models.
- Human vs AI: All games are validated to be beatable by humans on their first attempt. However, despite recent improvements, frontier models still struggle with surprisingly simple problems in the highest difficulty tiers.
- Leaderboard: The platform compares win rates across major models and agentic frameworks (e.g., Opus 5, GPT-5.5, Gemini 3.1 Pro).
Related event: DiG-bench: 70 Text Games Expose Frontier AI's Discovery Gaps(14 posts)→
More from coding & agent
- DIRECT: retrieval-based next-question suggestions for conversational agents — burkov · 2026-10-16
- Claude Code and Codex now autocomplete the user's next action — tokenbender · 2026-10-16
- Hermes Agent ships official Reverse Engineer Anything plugin with one-command setup — Teknium · 2026-10-16
- MLExpert: 35B MoE Agent Trained to Autonomously Handle Machine Learning Engineering — Shangheng Du · 2026-10-16
- Cowork-Gym: Alibaba Synthesizes 64.9K Agent Work Environments, Boosting Qwen by 8.7 Points — Anni Zou · 2026-10-16
- MMSearch-OPD: Joint RL and On-Policy Distillation for Long-Horizon Multimodal Search Agents — PekingUniversity · 2026-10-16