Gemini 4 Neon tops Bug Hunt bench, fixing 63/105 real repo bugs to edge out GPT-6.2 Astra
PawelHuryn · x · 2026-11-19
Pawel Huryn ran the Bug Hunt Bench leaderboard on 2 real repos with 105 planted bugs, scoring only verified fixes of planted defects.
- Gemini 4 Neon (xhigh): 63/105, the new #1, at $46.34 and 75.9 min per run
- GPT-6.2 Astra (max): 62.7, nearly tied; its xhigh tier scored 60.2
- Anthropic's Fable 5.2 (max) hit 58 but cost $139.65; Sonnet 5.5 (max) scored 51.3 while costing $154 and taking 2871 minutes — the priciest, slowest run
- Best value: GPT-6.2 Sol (54.7 for $7) and DeepSeek V4.1 Flash 1017 (42.7 for $1.84)
More effort tiers for Neon (medium, low) are promised in a follow-up thread.
More from coding & agent
- NormGym benchmark shows agents learn from own norm violations but not from others — xuanalogue · 2026-11-20
- MAC Bench: 8-agent teams beat solo agents via headcount, not coordination — gregd_nlp · 2026-11-20
- Every adopted 'compound engineering' company-wide so its codebase gets easier to maintain — every · 2026-11-20
- Start custom LLMs at 30B: runs on a single GPU and works with LoRA — ziv_ravid · 2026-11-20
- Not every agent task needs a general LLM — specialized small models by task type — ziv_ravid · 2026-11-20
- Before fine-tuning, distill traces into skills in-context to test if training is worth it — ziv_ravid · 2026-11-20