PLE: Physical-Engineering's Last Exam
Rujun Han, Yufan Zhuang, Arijit Ray, Han Yu, Zoey CuiZhu, George Lee, Yifei Ming, Binz Roy, Varun Sakalkar, Vishy Tirumalashetty, Burak Gokturk, Tomas Pfister, Chen-Yu Lee
cs.AI
2026-10-22
PLE scores agents on 85 billable electrical, civil, structural and mechanical jobs. Claude Opus 5.5 in Claude Code averages 50.6%, at 27M tokens and 63 minutes a task.
Frontier models now clear 90% on popular knowledge tests. Engineering benchmarks are still mostly textbook questions, single tool calls, or toy designs that fit in the prompt. Leading models already exceed 80% on EngiBench Level 2, and the best agent beats the Frontier-Eng baseline on 97.9% of that suite. A high score there still does not yield a billable package a licensed reviewer would sign.
Physical engineering is constrained by physics and by codes that have legal force, and the drawings need a licensed signature. The American Society of Civil Engineers puts the 2024-2033 U.S. infrastructure funding gap at $3.7 trillion. U.S. engineering-services firms report more than $300 billion in annual revenue. GDPval and APEX-Agents grade professional work by expert comparison or by rubric. In this field that is a weak proxy for value: two feeder schedules can pass every code check while one costs more than 20% extra to install.
PLE holds 85 long-horizon tasks: 30 mechanical, 27 electrical, 18 civil, and 10 structural. The brief reads like a note from a colleague. The sandbox contains PDFs, spreadsheets, DXF/DWG/IFC drawings, and mail, 5,170 files in all, 60.8 per task on average and 207 at the maximum. The required output is a memo, a calculation workbook, a schedule, or a drawing.
Forty-four tasks were written end to end by practising engineers hired through Mercor. They share 12 environments, with 62.8 files and about 3.0 MB each, almost all documents. Grading is a weighted binary rubric: 941 items, 21.4 per task. Load-bearing numbers carry weight 10 and formatting items weight 1. An LLM reads the delivered file. One brief asks for a Section 2 subtotal corrected to $1,303,717.70 in a memo ready for signature, with the figures scattered across an estimate and the drawings.
The other 41 start from an engineer's seed brief and a Creator-Solver loop. The Creator, Claude Opus 5 running in Claude Code, builds the corpus, an executable grader, and a reference deliverable from redistributable codes and drawings. The Solver, the same model in Hermes, makes three blind attempts with no grader access. If the mean comes in too high, constraints are added or the file set is thickened. A licensed engineer then checks the task and may harden it further. These environments average 58.7 files and 38.1 MB, 12.6 times the expert-authored set on disk, and they include distractors such as superseded revisions. Deliverables rise from 1.1 files to 4.9. All ten structural tasks come from this route.
Most co-authored tasks score as s = 0.4p + 0.6GO'. Here p is the fraction of constraints passed. G is a hard gate on safety and feasibility: if any hard check fails, G is 0, the economic term drops out, and the score falls back to 0.4p. O' is the lift over a naive design that is already compliant. On minimum-cost tasks, O = cbound / c, where cbound is a certified lower bound on the cost of any compliant design. Mere compliance earns no objective credit. The 0.6 weight stops an unsafe design from buying score by being cheap.
MEP means coordinating mechanical, electrical, and fire-protection routes in one space. One equipment-hall task has 8 input files and 128 cabinets. The agent must route cable tray, busway, chilled water, and sprinklers with zero clashes and every cabinet served. Two site-design tasks and two MEP tasks add an all-or-nothing gate: a hard clash or an unserved row scores 0% on MEP, and an infeasible site design is capped at 50%.
The eval crosses 14 models with seven harnesses, the tool loop around the model. GPT 6 Astra runs only in Codex, and Grok 4.7 only in Grok Build, so the grid is 86 pairs. Each attempt has a 6-hour budget. Grading happens outside the container. Every task is run three times. Expert rubrics are judged by Claude Opus 5.5. A public set of 29 tasks is planned for release with graders and the harness. The other 56 stay private for an open competition and leaderboard.
Claude Opus 5.5 in Claude Code leads, at 50.6% and $17.18 per task. The median task scores 43.5%. Forty-eight of the 85 tasks fall under 50%, and picking the best harness for each task still leaves 35 under 50%. Under this pair, two tasks score 0% on all three runs and seven score above 90%. Civil is the easiest at 69.9%, then electrical at 50.7%, structural at 46.6%, and mechanical at 40.5%. Expert-authored tasks score 53.1% and co-authored tasks 48.0%. Co-authored is lower in all seven harnesses for this model.
| Pair | Score | Cost per task |
| Claude Opus 5.5 + Claude Code | 50.6% | $17.18 |
| Claude Opus 5.5 + OpenHands | 49.9% | $14.82 |
| GPT 6 Astra + Codex | 48.3% | $9.15 |
| Gemini 4 Argon + Claude Code | 43.2% | $8.21 |
| Gemini 3.8 Flash + Claude Code | 28.2% | $2.55 |
GPT 6 Astra trails by 2.3 points at about half the cost. Gemini 3.8 Flash keeps about 56% of the leading score at about 15% of the cost. Further down, Claude Sonnet 5.5 in Antigravity scores 45.2%, the best Claude Opus 5 configuration scores 42.6%, Grok 4.7 in Grok Build scores 32.0%, and Gemini 3.1 Pro scores 14.2%. Qwen3 Coder 480B, MiniMax M2, Kimi K2 Thinking, and Llama 4 Maverick all stay under 7%. Llama 4 Maverick in Claude Code scores 0.4%.
The model moves the score more than the harness. On Opus 5.5 the seven harnesses span only 47.3% (Pi) to 50.6% (Claude Code). For a fixed model the harness range is 0.7 to 12.1 points, median 3.0, against about 50 points between models. Spend does not follow score. The same Opus 5.5 reaches 49.3% in Codex at $35.15 and 47.4% in OpenCode at $12.53. The eight most expensive configurations, all Claude models, cost $27 to $54 and score 33.8% to 49.3%, every one of them below the $17.18 leader.
That leading pair averages 27 million tokens and 63 minutes per task. Co-authored attempts take 39.3 million tokens and 97 minutes. Expert-authored attempts take 15.6 million and 31 minutes. Only 11 of 255 runs score 0%. On co-authored work, delivered items pass 91% of constraints on average, yet only 41.3% clear every hard constraint and so earn the objective term. Across all 86 pairs, 23.2% of runs score 0%. Among those zeros, 25.3% have no gradable file, 4.3% hit a timeout or a crash, and 70.4% return files the grader scores at zero.
The rubric judge was re-run on 1,159 LLM-judged runs of Claude Opus 5.5. A second Opus 5.5 pass matches 98.7% of item verdicts (kappa 0.97) and leaves the mean unchanged. Claude Opus 5 agrees on 95.8% and scores 3.3 points higher (about 2.1 on the full benchmark). Gemini 4 Argon agrees on 95.5% and scores 1.3 higher (about 0.9 overall). The lift sits almost entirely on rubric tasks. Claude Code ranks first under all three judges. On the 44 rubric tasks in Claude Code, Gemini 4 Argon adds 3.6 points where a second Opus 5.5 pass adds 1.3. The paper calls that 2.3-point gap combined self-preference. Set-level means are stable: the three Claude Code passes are 52.0%, 49.3%, and 50.6%. Single tasks are not. The average range over three runs is 15.4 points, and 105 of 595 cells span at least 30.
PLE is a measurement, not a stronger design algorithm. The tasks, corpora, and graders hold the best pair to 50.6%, and they locate the miss: items satisfy 91% of constraints, while only 41.3% clear every hard constraint at once. The hard part is being fully safe and feasible while pushing cost toward the bound.
Changing the model spreads scores by about 50 points. Changing the harness moves a typical model by about 3, while the bill for a similar score can more than double. A PLE number needs both the model and the harness. The public 29 tasks are enough for a regression check. A full board, at 63 minutes and 27 million tokens a task, is an expensive rerun.
The rubric judge is Claude Opus 5.5, the model in first place. Twelve co-authored graders also send three to six narrative items to that same model. Absolute scores rise under the other judges they tried, while Claude Code stays first. The headline 50.6% averages LLM rubric points together with a programmatic economic score. Those are different instruments.
Difficulty was calibrated with Claude Opus 5 in Hermes. Tasks the solver still cleared were tightened, which plugs shortcuts that one model found. Opus 5.5 still finishes first. The task shape was fit to a Claude solver anyway, and the difficulty threshold is not reported apart from the models under test.
No licensed engineer sat the same graders. Authoring an expert task paid $4,663 on average, and validating plus hardening a co-authored task paid $3,157. That prices human time. The score a signing engineer would post is still missing. The reported gap is to a perfect grader.
The 44 expert-authored tasks share 12 environments, so they are not independent draws. Per-task ranges are wide enough that task-level ranks are noise. GPT 6 Astra and Grok 4.7 each ran in a single harness. The paper shows no sensitivity check on the 0.4/0.6 weights or on how tight cbound is. All-or-nothing gates on site design and MEP turn a near miss into a zero. Environments were written from scratch or taken from redistributable public sources. They are not live jobs on which a firm carries liability.