Cheap builder + frontier reviewer loop matches flagship on 66 tasks at ~1/16 the cost ($0.05 vs $0.72)
KrystalUnity · reddit · 2026-11-11
The author proposes the Krystal Loop Protocol: a cheap model writes code in a sandbox, tests run, then a frontier model from a different vendor reviews the exact diff. Reviews cost 1.5 cents each because the reviewer only sees a small packet—task, diff, test output. The key rule: the reviewer can't just say "this looks wrong"—it must write a test proving the bug. Failing test = real bug, gets fixed; passing test = complaint discarded. Fixes get re-checked and re-reviewed until clean or a limit is hit.
Results on 66 CooperBench feature tasks, graded by hidden tests:
- gpt-6-astra alone: 56 solved, $0.721/task
- gpt-6.1-sol alone: 56, $0.123
- deepseek-flash alone (3 runs): 54–57, $0.014–0.027
- Loop (deepseek-flash builds, gpt-6.1-sol reviews, 3 runs): 54–59, $0.046–0.061
Honest caveats: the cheap model alone was closer than expected—the review added only 0–2 solved tasks per run. The win is mostly price, not score: in the best run the review caught 77 real bugs proven by failing tests (70 fixed) and never broke working code.
The loop mattered more for conflicting features: on 30 feature pairs, plain git merge worked on 2; two agents handing off via short structured messages got 28 working at 10 cents/pair (chat-based collaboration also hit 28 but burned 581 model calls vs 333); one frontier model building both from scratch managed 22 at $0.196—nearly double the cost.
The reviewer can also be a CLI subscription you already pay: Gemini, Kimi, Grok and Qwen reviewed identical output and landed within two tasks of each other; Gemini matched the metered frontier critic exactly (56) at under 3 cents/task in API-equivalent spend.
Everything recomputes from public receipts with no API keys (python3 -m bench.verify); rerunning the loop costs $3. Open questions the author wants grilled: is 66 tasks enough, will it break on real repos, and could a decent local model as builder hold the score near zero cost.
More from coding & agent
- Amazon paper: single agents beat multi-agent setups on enterprise SOPs at 1.4–4.7x lower cost — rohanpaul_ai · 2026-11-11
- Amazon paper: multi-agent setups burn 1.4-4.7x tokens for small gains; single agent is the default — rohanpaul_ai · 2026-11-11
- Dev shows off building a retro FPS entirely via Codex Mobile — 503 tests passing — Dimillian · 2026-11-11
- Explosion launches Ellf directory: 52 NLP templates plus multi-agent annotation workflow — _inesmontani · 2026-11-11
- Meme: 'Stochastic AI Agent Architecture' Is Just a Markov Chain in Disguise — PrasoonPratham · 2026-11-11
- OpenCode adds Gemini 4 Carbon and Claude Fable 5.2 support — dean_rie · 2026-11-11