DiG-bench Released: New Text-Based Benchmark Stumps Frontier LLMs on Simple Discovery
misovalko · x · 2026-08-12
Researchers from Princeton, MIT, and other institutions have introduced DiG-bench (Discovery in Games), a new text-based benchmark for in-context discovery.
Benchmark Mechanics
- The benchmark features novel text-based concepts and mechanisms where players and models must figure out the rules through interaction with very little prior instruction.
- The design draws inspiration from early tests of abstraction and analogy by Doug Hofstadter and Melanie Mitchell.
Key Findings
- While frontier models have improved significantly in recent months, they are still stumped by surprisingly simple discovery problems within their native text domain that humans can easily solve.
Related event: DiG-bench: 70 Text Games Expose Frontier AI's Discovery Gaps(14 posts)→
More from Research
- DIRECT: retrieval-based next-question suggestions for conversational agents — burkov · 2026-10-16
- AI-Driven Timing-Aware Power Optimization Uses Learned Ranking With Baseline-Preserving Repair — Amit Sharma · 2026-10-16
- Waypoint-OPD: Alibaba Gates On-Policy Distillation via Early-Late Teacher Shift Agreement — Bingxi Hou · 2026-10-16
- MJev-Spatial: Single-Pass Classification Beats Autoregressive VLMs on Spatial Reasoning — mindlab-research · 2026-10-16
- MLExpert: 35B MoE Agent Trained to Autonomously Handle Machine Learning Engineering — Shangheng Du · 2026-10-16
- VLA-Δ: Zhejiang University Adds Physical State Transition Supervision to VLA Models — OmniAI-ZJU · 2026-10-16