Sudo L7 benchmark uses 76-word natural prompts vs 291-427 words for rivals
echen · x · 2026-10-23
HelloSurgeAI explains sudo L7 was deliberately designed with natural, short prompts — 76 words median vs 291-427 across DeepSWE, FrontierSWE and SWE-Marathon. An L7 prompt is just 'Fix the threads page search, it crashes in dev mode', while L3s get 400-word specs from PMs and L7s are told 'make that faster' and must know when to ignore the PM. The sample SWE-marathon prompt (shown in full) specifies a Tabula spreadsheet product: REST contract, Excel-365-compatible formula engine with LET/LAMBDA and dynamic arrays, error sentinels and a full function catalog — testing whether agents can handle decisions engineers make when instructions aren't spelled out, like 'Can we safely cache this?'
More from coding & agent
- Grok Build ships v1.0.66/67: @ file completion 8x faster, worktree approvals — mark_k · 2026-10-23
- Prime Intellect lays out continual learning roadmap: mining agent exhaust for self-improving models — willcb · 2026-10-23
- Claude spent 2 days reimplementing vLLM — readable code review reveals a maintainability nightmare — bingxu_ · 2026-10-23
- Percy Liang teaches new MIT course on self-evolving AI agents across four evolution levels — pliang279 · 2026-10-23
- Using GrokBot as an OSINT Tool to Trace the Origin of Forged AfD Emails — FlorianGallwitz · 2026-10-23
- intentoo: an MCP server where AI agents negotiate matches before humans ever meet — Appropriate-Ease2981 · 2026-10-23