Google and Mercor's PLE benchmark: all frontier agents score below 50% on 85 real engineering tasks
teortaxesTex · x · 2026-11-04
Google, talent company Mercor, and academic researchers (Jiajun Wu, Saining Xie, and others) released Physical-Engineering's Last Exam (PLE), the first benchmark testing AI agents end to end on real-world engineering work.
Key points:
- 85 expert-reviewed long-horizon tasks across structural (10), civil (18), electrical (27), and mechanical (30) engineering, spanning calculations, drawings, and design reviews
- On the private test set every frontier agent scored below 50%: Claude Opus 5.5 leads at 48.3%, followed by GPT-6 Astra (45.8%), Claude Sonnet 5.5 (43.5%), Gemini 4 Argon (40.6%), Claude Opus 5 (40.0%)
- A steep tail: GLM 5.2 at 25%, DeepSeek-V4.1-Flash 14.4%, while Kimi K2 Thinking, MiniMax M2, Qwen3 variants and open small models score in single digits
- On co-authored tasks the top agent satisfied 91% of individual constraints, but overall completion remains limited
Commenter teortaxesTex calls it "same picture, even worse," underscoring the ceiling on hard engineering tasks.
More from Models
- Dev's 4 days with Codex: tool admits 16 failures, Claude had to finish the job — v0id_user · 2026-11-04
- shadcn: Opus's 'taste' can't be benchmarked overnight, it takes real product building — shadcn · 2026-11-04
- Users say Claude's Ultrafast mode burns limits 8x faster — a 'flex' speed tier would make more sense — Lucky_Creme_5208 · 2026-11-04
- OpenAI quietly reverses its reverse engineering ban, diverging from Anthropic's stricter stance — AccBalanced · 2026-11-04
- Built with Claude Haiku 5.5, not Opus: pixel-art animation shows small models often suffice — Lazar988 · 2026-11-04
- Mistral Large 4 tested on real Django CVE: 133 agents find, prove and patch the bug — MistralAI · 2026-11-04