Google's PLE benchmark: best agent scores just 50.6% on real physical engineering tasks
rohanpaul_ai · x · 2026-10-29
Google's PLE (Physical-Engineering's Last Exam) paper argues existing engineering benchmarks are saturated textbook/toy problems far from real engineering work.
- The benchmark has 85 long-horizon agentic tasks across electrical, civil, structural, and mechanical engineering, authored or checked by practicing engineers; each task requires reading 61 project files (PDFs, spreadsheets, CAD) and delivering schedules, reports, or 2D/3D drawings.
- Claude Opus 5.5 in Claude Code topped 14 models at 50.6%, spending 27M tokens and 63 minutes per task; its designs met 91% of constraints but only 41.3% satisfied every hard constraint — failures mostly from missing safety or feasibility rules.
- Advice for agent builders: gate every design on hard safety/code limits, and prioritize model choice over tuning the agent scaffold.
More from coding & agent
- Celesto AI lets you run GitHub Actions on its cloud computers with per-job pricing — aniketmaurya · 2026-10-29
- GitHub sees 7.4B monthly commits from AI agents, adds 4M CPU cores to cope — ai · 2026-10-29
- 5 practical tips to cut CI time when coding with AI agents — aniketmaurya · 2026-10-29
- Agent rate-limited by Greenhouse reads the email security code itself: "I already have it" — arthurcolle · 2026-10-29
- Omnigent v0.20.0 ships in-tab PR merging, Databricks managed sandboxes, and /mcp command — matei_zaharia · 2026-10-29
- "Paste your latest prompt": sharing an aggressive multi-thread agent prompt — gethackteam · 2026-10-29