50-PR Benchmark: GPT-6 Astra Finds 92 Bugs, GPT-5.6 Luna Matches 75% at 3.6% of the Cost
entelligenceai17 · reddit · 2026-09-14
Entelligence ran a code-review benchmark on 50 real PRs from Cal.com, Sentry, Discourse, Keycloak, and Grafana:
- GPT-6 Astra confirmed 92 bugs vs 69 for GPT-5.6 Luna
- Yet Luna caught 75% of confirmed bugs at just 3.6% of the cost
- Full breakdown by bug class (data/logic, security, concurrency), precision, latency, and avg output tokens; findings independently verified
- Next up: an Astra vs Fable 5.1 benchmark, with the team soliciting eval feedback
Takeaway: flagship models win on recall, but cheaper models are extremely cost-effective for code review.
More from coding & agent
- LangChain CEO opens Interrupt London with keynote on why orgs should own their intelligence — LangChain · 2026-10-30
- Simlock: open-source control plane gives parallel coding agents their own iOS/Android simulators — Scobleizer · 2026-10-30
- YC experiment: Claude agents formed democracies while GPT agents formed hives — soumitrashukla9 · 2026-10-30
- Keyfleet launches open canvas where personal AI agents team up with a shared USDC treasury — seanwbren · 2026-10-30
- How Oxide Computer uses LLMs internally — FAQ Friday #66 — Sethwinterroth · 2026-10-30
- Clear Thought MCP Server adds structured reasoning and mental models to AI clients — modelcontextprotocol · 2026-10-30