Half of your coding agent's success is luck: randomness hits 32-52% across models
micahgoldblum · x · 2026-10-15
Everyone codes with agents now, but a circulating analysis claims 50% of an agent's success comes down to luck. Shared figures show the random share of successful runs at 52% for Claude Fable 5, 50% for Claude Opus 5, and 32% for GPT-6 Astra — implying single-run benchmark gaps may be mostly noise, and evals need repeated trials rather than one-off scores.
More from coding & agent
- Dev vibe-codes open-source newsletter platform, 83% cheaper than Beehiiv from $19/mo — marclou · 2026-10-15
- ERP AI launches Proto: 'headless SaaS' business software built for AI agents — Aiden_Tech_Ai · 2026-10-15
- FLUX.3 Image Klein ships open in Diffusers: layout control, 2s per image on H100 — ariG23498 · 2026-10-15
- Engineering managers need deep reading, not just agent summaries: 5 recommended resources — rseroter · 2026-10-15
- Adding motion graphics with just a prompt: the Claude Motion x ElevenLabs MCP workflow — CurieuxExplorer · 2026-10-15
- A $15k AI SEO client-finding system built on Claude, free to download — aryanXmahajan · 2026-10-15