DiG-bench: Evaluating AI Discovery via 70 Hidden-Rule Text Games
jcrwhittington · x · 2026-08-12
To evaluate AI's genuine ability to explore and make scientific discoveries in unknown environments, researchers introduced DiG-bench. The benchmark consists of 70 self-contained text-based discovery games, each rendered as a minimal string where the AI acts as an arrow navigating the world.
Key features include:
- Hidden Rules: The rules and win conditions are completely concealed, forcing the AI to rely on real experimentation and exploration to progress.
- Calibrated Difficulty: Features 7 difficulty tiers aligned with frontier models. The games are hard but beatable, with every game having been solved by at least one human on their first try.
- Secure Evaluation: 21 games are public, while 49 remain private to prevent contamination and ensure secure evaluation.
The author argues that for AI to chart new science rather than just combining known tools, it must first prove it can make discoveries in the easiest possible settings.
Related event: DiG-bench: 70 Text Games Expose Frontier AI's Discovery Gaps(14 posts)→
More from Research
- DIY LEGO bench tests AI brick-building, GPT-6 Astra beats Opus — TheLegoless · 2026-10-16
- DIRECT: retrieval-based next-question suggestions for conversational agents — burkov · 2026-10-16
- AI-Driven Timing-Aware Power Optimization Uses Learned Ranking With Baseline-Preserving Repair — Amit Sharma · 2026-10-16
- Waypoint-OPD: Alibaba Gates On-Policy Distillation via Early-Late Teacher Shift Agreement — Bingxi Hou · 2026-10-16
- MJev-Spatial: Single-Pass Classification Beats Autoregressive VLMs on Spatial Reasoning — mindlab-research · 2026-10-16
- MLExpert: 35B MoE Agent Trained to Autonomously Handle Machine Learning Engineering — Shangheng Du · 2026-10-16