SciML bench: open-weight models miss all bugs, approve broken code; frontier models shine
ChrisRackauckas · x · 2026-10-18
Chris Rackauckas shares interim SciML Reviewer Bench results: 24 real SciML pull requests (17 with defects confirmed by an executable reproducer, 7 clean) given to AI reviewers who checkout the code, run tests, and issue verdicts.
Key findings:
- Frontier models lead: gpt-6.1-sol and gpt-6-astra had 0% harmful verdicts, catching 6/7 and 5/7 bugs with zero broken-code approvals; gpt-6-luna caught 0/3 and approved all 3 broken PRs.
- Open-weight models largely fail: most show 60–100% harmful verdict rates. DeepSeek V4.1-Flash, Qwen3.8-Max, GLM-5.3 and Kimi K3 caught zero bugs while approving many defective PRs; Inkling and MiniMax M3 approved 5/5 broken PRs.
- Conclusion: code review still requires frontier models — top open-weight models aren't close in head-to-heads. The bench runs daily with ongoing updates.
More from coding & agent
- Dev builds Arch-based OS with custom desktop and browser using Claude Code — Snoo-10493 · 2026-10-18
- Pi Durable went through 7 rewrites yet agent session files still lack a standard — ricklamers · 2026-10-18
- Five SaaS agent workflows fail the same way — and it's never model intelligence — usehive · 2026-10-18
- Claude skill generates hand-drawn interactive diagrams that teach kids how things work — CatAstro_Piyush · 2026-10-18
- SpudStack on Grok Bot marketplace: a rigor reviewer for coding agents — alexcovo_eth · 2026-10-18
- OSS AI search XERJ hits 2,000 daily installs in 3 months, GitHub Rust #1 — alexcovo_eth · 2026-10-18