Google's Deep Review tore a 35-page benchmark spec apart, finding 36 flaws
ChrisUniverse · x · 2026-10-18
A developer tried Google DeepMind's new Deep Review expecting generic AI peer-review fluff, but instead found it behaved like a hostile NeurIPS chair on his 35-page private benchmark spec:
- Derived a 9x–27x penalty asymmetry between linear and exponential terms, causing rank reversals
- Proved a pigeonhole flaw in sampling rules (30 runs < 40 needed across 2 exclusive rates)
- Caught a zero-variance bootstrap bug and a 97.5th vs 95th quantile mismatch
- Flagged TOCTOU race windows during sandbox teardowns
Total: 36 spec flaws — 6 Critical, 14 Major, 10 Minor, plus 6 suggestions. His takeaway: everyone tests code, but nobody red-teams broken math the way Deep Review does.
Related event: Hands-on: Google Deep Review Spots 36 Flaws in 35-Page Spec(2 posts)→
More from coding & agent
- Papers with Code adds Auto-Research category, 166 papers already tracked — NielsRogge · 2026-10-18
- Claude Code shows fake summarized thinking as assistant messages — ~37% of transcript — Sauers_ · 2026-10-18
- Dev builds a demon-summoning TUI for managing AI agents — voidwalker_com · 2026-10-18
- Fix wandering coding agents with a DOMAIN_INVARIANT.md file of domain rules — 0xkarasy · 2026-10-18
- Inside NVIDIA's Build-a-Claw Berlin: agents everywhere plus Reachy Mini robots — Hesamation · 2026-10-18
- Reason Network launches: free open-source P2P engine lets Claude Code, Codex and other agents talk — TianbaoX · 2026-10-18