Google's PLE benchmark: best AI agent scores only 50.6% on professional engineering deliverables
dair_ai · x · 2026-10-24
Google Cloud AI Research introduces PLE (Physical-Engineering's Last Exam), an agentic benchmark of 85 long-horizon tasks across electrical, civil, structural, and mechanical engineering, where solving means producing a billable deliverable a licensed reviewer would sign off on. Tasks come via expert-authored and adversarial Creator-Solver co-authored pipelines. Under seven public agent harnesses, the best configuration averages just 50.6%, spending 27M tokens and 63 minutes per task. 29 of 85 tasks will be released publicly, the rest kept for an open competition and leaderboard.
More from coding & agent
- Claude Code Routines runs overnight, screens 380 clinics into 30 outreach drafts — Arindam_1729 · 2026-10-24
- Factory and Anthropic share customer story on long-horizon coding agents and 'dark factories' — matanSF · 2026-10-24
- Dev argues Rust rewrite of Apache2 with Verus spec beats running C code in expectation — jd_pressman · 2026-10-24
- Agent-designed specs tested against every historical CVE, then adversarially attacked — jd_pressman · 2026-10-24
- Carmack amplifies Goose: a memory-safe systems language that beats C++ and Rust without a heap — ID_AA_Carmack · 2026-10-24
- Scoble points a Grok Bot at 20,000 curated accounts to build his personalized AI news feed — Scobleizer · 2026-10-24