Surge AI's sudo L7 benchmark: every Claude beats top GPTs on staff-level engineering judgment
echen · x · 2026-11-17
Surge AI's sudo L7 benchmark measures staff-level engineering judgment rather than unit-test passing. At max reasoning effort, every Claude outscored the top GPTs—even Haiku 5.5 beat GPT-6.1 Astra and Sol. Key case: rolling a customer-specific feature to a second customer, GPT-6.1 Sol copied the MVP implementation and missed that it would leak customer one's sensitive data to customer two; Haiku caught the security risk. echen frames it: benchmarks look for unit-test-passers, sudo L7 looks for jeff deans—the frontier is shifting from correct code to understanding what users actually need.
More from coding & agent
- Early hands-on with Gemini 4 Carbon in Antigravity CLI: slow but insightful — iskander · 2026-11-17
- Clinical genomics agent spots two lethal translocations NGS missed, in first peer-reviewed paper — danielmckinn0n · 2026-11-17
- Tsinghua's ActionOPD lifts Qwen3-4B agent success on τ²-bench from 29.2% to 40% — jiqizhixin · 2026-11-17
- Anthropic's daily code merges grew 12x in 4 months, now bottlenecked by human-scale infra — Dr_Singularity · 2026-11-17
- Kanban is a dead end for AI agents, argues dev pushing live reasoning-trace monitoring — kevinkern · 2026-11-17
- Software factory practice: verification infrastructure beats prompt and model choice — vikvang1 · 2026-11-17