E2E-SWE benchmark: 186 tasks test coding agents building repos from scratch, best pass@1 just 67.7%
jyangballin · x · 2026-10-30
- Hantian Ding et al. released E2E-SWE, a benchmark of 186 whole-repository generation tasks across 11 languages testing whether coding agents can build complete, installable software repos end to end.
- Setup: agents get only a natural-language specification and an empty workspace, work offline, and succeed only if the project installs and passes all hidden functional tests; tasks are co-developed by engineers and LLMs and audited by autonomous agents.
- Results: across 13 frontier models, pass@1 ranges from 11.7% to 67.7%, showing strong differentiation. Paper and code are public.
- John Yang notes that different whole-repo-generation setups (executable vs spec) lead to markedly different agent iteration behaviors.
More from coding & agent
- Building a Black Friday deal-hunting agent: agentic commerce starts with discounts — jeff_weinstein · 2026-10-30
- Sentry CEO on agent determinism: why system prompts are almost never the answer — zeeg · 2026-10-30
- Claude Opus autonomously builds a walkable 3D IKEA room from product photos via HF agents — victormustar · 2026-10-30
- claude-obsidian hits 16k stars: Claude Code auto-builds and maintains a linked Markdown knowledge vault — JeremyCMorgan · 2026-10-30
- openrock launches: one open-source chat app for every AWS Bedrock model — maelswarm · 2026-10-30
- Arc: a 70 MB streaming speech model that does ASR and entity linking in one step — Working_Hat5120 · 2026-10-30