Google's PLE: 85 real engineering tasks, best agent config scores only 50.6%
rohanpaul_ai · x · 2026-10-29
Google Cloud AI Research released PLE: Physical-Engineering's Last Exam on arXiv, the first benchmark testing whether LLM agents can do real, billable physical engineering work.
- Tasks: 85 long-horizon agentic tasks across electrical, civil, structural, and mechanical engineering; each is a realistic deliverable requiring modeling physics and legal-code requirements over multi-format corpora (PDFs, spreadsheets, CAD), optimizing material and installation costs, and producing 2D/3D design packages a licensed reviewer would sign off.
- Curation: Expert-Authored tasks are built end-to-end by practicing engineers; Co-Authored tasks use an adversarial Creator–Solver agent pipeline to construct project environments, executable graders, and reference deliverables, validated and difficulty-intensified by experts.
- Results: Across seven public agent harnesses, the strongest configuration averages just 50.6%, spending 27M tokens and 63 minutes per task — a substantial gap from autonomous physical engineering design.
- Release plan: 29 of 85 tasks will be public with graders and harness; the rest are held for an open competition and leaderboard.
More from coding & agent
- Celesto AI lets you run GitHub Actions on its cloud computers with per-job pricing — aniketmaurya · 2026-10-29
- GitHub sees 7.4B monthly commits from AI agents, adds 4M CPU cores to cope — ai · 2026-10-29
- 5 practical tips to cut CI time when coding with AI agents — aniketmaurya · 2026-10-29
- Agent rate-limited by Greenhouse reads the email security code itself: "I already have it" — arthurcolle · 2026-10-29
- Omnigent v0.20.0 ships in-tab PR merging, Databricks managed sandboxes, and /mcp command — matei_zaharia · 2026-10-29
- "Paste your latest prompt": sharing an aggressive multi-thread agent prompt — gethackteam · 2026-10-29