200-repo eval: only frontier models like Opus are reliable for code review
ChrisRackauckas · x · 2026-10-18
SciML maintainer Chris Rackauckas shares eval results from running coding agents across 200 repos (26k stars) in the Julia/SciML org: code review requires frontier models to be correct — Luna/Sonnet/SWE2-class models can write code, but reviewers must be Opus/Sol-class, making agent usage bottlenecked on review even though writing dominates. He's scanning all models on Fireworks for cheaper alternatives, but nothing beyond Sol and Opus looks promising so far.
More from coding & agent
- Managed AI agent platforms everywhere — real demand or a solution in search of a problem? — Ok_Philosophy_4031 · 2026-10-18
- Chris Rackauckas: fine-tuned local models run robot heads on Orin 64GB, Opus writes the scripts — ChrisRackauckas · 2026-10-18
- Open models outperform on routine code maintenance, but only Opus finds deep defects — ChrisRackauckas · 2026-10-18
- Open-source ComfyUI storyboard and video prompt generator for MiniMax H3 workflows — Resident-Gain-5424 · 2026-10-18
- Concentrate AI reveals ~$70k/month bill for AI coding subscriptions and API calls — ChrisRackauckas · 2026-10-18
- Pi Durable Agent harness on Cloudflare powers fully autonomous apps with zero human input — MicahBerkley · 2026-10-18