DiG-bench Results: Closed-Source Models Lead by Leaps, Agent Frameworks Offer No Benefit
jcrwhittington · x · 2026-08-12
The research team detailed the models' specific performance on the DiG-bench:
- Significant Model Gap: Closed-source models like Opus 5 and Fable showed real capability increases but still struggle with the hardest Tiers 6 and 7. There is a massive gap between closed-source (Opus 5) and open-source (Kimi K3), and an even larger gap to smaller open-source models (Qwen 3.6 27B).
- No Benefit from Agent Frameworks: Using agentic harnesses like Prime offered no benefits over a basic, non-coding harness.
- Rule Discovery is the Bottleneck: When ground-truth rules are given in plain language, even the weakest models score near perfect, proving that application isn't the issue, but rather autonomous rule discovery.
Related event: DiG-bench: 70 Text Games Expose Frontier AI's Discovery Gaps(14 posts)→
More from Models
- Unverified claim: Apple and Meta's 816B-parameter Muse Spark Open runs on two Mac Studios — TansuYegen · 2026-10-16
- Gemini 4 model card: model eliminated months of trial-and-error on bioweapons research — Puzzleheaded-King584 · 2026-10-16
- Every AI picks 7: Anthropic engineer's rule for escaping model taste convergence — rubenhassid · 2026-10-16
- Brain teaser: strip all attention, MLP and embedding matrices from a Llama 3 — how many trainable params remain? — Sauers_ · 2026-10-16
- Lean v4.35.0 ships verified kernels — one written and proven consistent by Claude — geoffreyirving · 2026-10-16
- One flagged seat froze an entire Claude team org — founders share backup plans — RexDouglass · 2026-10-16