Mercor launches OWE-Bench: 26 real engineering projects that stump coding agents
ben_burtenshaw · x · 2026-11-13
Mercor released OWE-Bench, a long-horizon engineering benchmark with 26 full projects designed by domain experts (master's/PhD, 10+ years experience), spanning 4 domains from GPU kernels to aircraft design. Attempts take 4–18 hours and 124M tokens on average, testing end-to-end project completion.
Top leaderboard (mean normalized score, 26 tasks, 8 trials per model):
- Claude Fable 5.2: 54.6%
- GPT-6 Astra: 44.2%
- Claude Opus 5.5: 41.6%
Other notables: GPT-6.1 Sol 41.2%, Claude Sonnet 5.5 39.0%, Gemini 4 Carbon 32.8%; Kimi K3.1 only 13.6%, DeepSeek V4.1 Flash 12.4%, GLM-5.3 11.8%, Grok 4.7 at 7.0%.
The benchmark is fully open source: 16 tasks ship with public graders you can run yourself, 10 keep private graders (including Design/Build/Fly Aircraft in a private flight simulator). Tasks are in Harbor format on GitHub, dataset on Hugging Face.
More from coding & agent
- Open-source flight search MCP server refuses to answer what it can't confirm, curbing model hallucination — Fast_Entrance7839 · 2026-11-13
- Amplitude launches Wave: AI that decides what to build next from product data — testingcatalog · 2026-11-13
- jxnlco: when AI makes you faster, go deep on new things and grow your taste — jxnlco · 2026-11-13
- Vibe-coded space-defense game ORBITAL SIEGE shows Three.js gameplay from one prompt — kimmonismus · 2026-11-13
- AI-generated WebGL2 shader bends light around a black hole on the first try — kimmonismus · 2026-11-13
- Sententos puts a personal agent in your Mac's notch that can take over your screen — ycombinator · 2026-11-13