Google DeepMind's Deep Review tears apart a benchmark spec, finding 36 flaws
ChrisUniverse · x · 2026-10-18
ChrisUniverse tested Google DeepMind's new Deep Review on a 35-page private benchmark spec and got far more than generic AI peer-review fluff: it found 36 flaws (6 Critical, 14 Major, 10 Minor, plus 6 suggestions). Highlights: a derived 9x-27x penalty asymmetry between linear and exponential terms causing rank reversals; a pigeonhole flaw in sample rules (30 runs < 40 needed across 2 exclusive rates); a zero-variance bootstrap bug plus a 97.5th vs 95th quantile mismatch; and TOCTOU race windows during sandbox teardowns posing real security issues. The reposter adds that the most dangerous benchmark defects live in the spec itself, not the code — audit the spec before paying to run models, and verify each fix with minimal counterexamples and independent review so AI improves evals' falsifiability rather than adding authority.
More from Models
- DeepSeek V4.1 Flash-1017 hits 77% Pass@1 at max effort, lower levels barely help — zainhas · 2026-10-18
- DeepSeek's new model spotted in eval circles, cyber benchmarks requested — scaling01 · 2026-10-18
- Correction: Claude's 'thinking blocks' are summarized user-facing messages, not reasoning — Sauers_ · 2026-10-18
- DeepSWE benchmark: DeepSeek V41 Flash-1017 tops chart at 77% pass rate with lowest cost — Immediate_Simple_217 · 2026-10-18
- Ex-Microsoft exec Parakhin: after 2 days with AI advisors, I still add value — MParakhin · 2026-10-18
- $20 plan showdown: user finds Codex far less token-efficient than Claude — cneuralnetwork · 2026-10-18