Bug Hunt Bench: frontier coding models graded on 105 planted real-world bugs, cost spread ~200x
PawelHuryn · x · 2026-09-24
Pawel Huryn (The Product Compass) launched Bug Hunt Bench, a hands-on leaderboard grading frontier coding models on 105 bugs planted in real repos.
- Blind grading: one prompt per repo, blind-graded scoring; supports comparing the same model across reasoning tiers and harnesses.
- Cost view: includes score-vs-cost and score-vs-time charts — cost spread is roughly 200x (log scale), while runtime spans under 7x (linear scale).
- Full run notes and caveats are public on GitHub (results/run-notes.md); the author promises hands-on, no-hype, run-first methodology.
More from coding & agent
- KAIST's WikiTrace: evidence-linked memory fixes agent "memory-induced degradation" — dair_ai · 2026-11-11
- Why inference speed will be AI products' biggest competitive edge in 1–2 years — itsOmSarraf_ · 2026-11-11
- What happened when I moved vector retrieval into Snowflake SQL instead of orchestrating RAG outside — dima806_dima · 2026-11-11
- OpenAI engineer runs Codex for 26 days straight hunting Atlantis, spawning "vibe archaeology" — Polymarket · 2026-11-11
- He replaced a $5,000/month ghostwriter with 11 Claude Skills and earned $47,000 from one post — aryanXmahajan · 2026-11-11
- Refero Styles ships 2,000+ DESIGN.md files to give Claude Code real design taste — bbssppllvv · 2026-11-11