Introducing DiG-bench: 70 Text Games to Evaluate AI Scientific Discovery
jcrwhittington · x · 2026-08-12
A research team officially introduced DiG-bench (Discovery Games Benchmark), a benchmark designed to test AI's capacity for scientific discovery.
Core Design:
- Features 70 new interactive text-based games requiring agents to discover and apply unknown rules via experimentation.
- The purely text-based environment eliminates visual confounds, directly testing LLMs' discovery capabilities in their native domain.
- Humans and frontier models are tested under identical interfaces, action sets, and step budgets.
Key Findings:
- Games are categorized into 7 difficulty tiers. All are beatable by humans on their first attempt, but the best AI models struggle significantly with the highest tiers.
- Closed-source models (e.g., Opus 5) significantly outperform open-source models (e.g., Kimi K3).
Related event: DiG-bench: 70 Text Games Expose Frontier AI's Discovery Gaps(14 posts)→
More from Research
- DIRECT: retrieval-based next-question suggestions for conversational agents — burkov · 2026-10-16
- AI-Driven Timing-Aware Power Optimization Uses Learned Ranking With Baseline-Preserving Repair — Amit Sharma · 2026-10-16
- Waypoint-OPD: Alibaba Gates On-Policy Distillation via Early-Late Teacher Shift Agreement — Bingxi Hou · 2026-10-16
- MJev-Spatial: Single-Pass Classification Beats Autoregressive VLMs on Spatial Reasoning — mindlab-research · 2026-10-16
- MLExpert: 35B MoE Agent Trained to Autonomously Handle Machine Learning Engineering — Shangheng Du · 2026-10-16
- VLA-Δ: Zhejiang University Adds Physical State Transition Supervision to VLA Models — OmniAI-ZJU · 2026-10-16