EngiWorld: What Can Frontier Agents Deliver in Professional Engineering Environments?
Hongcheng Gao, Hailong Qu, Yu Lei, Henghui Sun, Haoyang Li, Yipeng Wei, Naihao Xue, Xiaohan Yu, Zhuo Tao, Yihe Zang, Yajiao Wang, Jingyi Tang, Yi Li, Jingjing Zhou, Jie Luo, Bohan Zeng, Chengyu Shen, Hao Jiang, Chong Chen, Bowen Qu, Olive Huang, Zeqiang Wang
cs.AI, cs.CL
2026-09-29
EngiWorld puts agents on 1,301 expert tasks across 26 professional engineering apps. Claude Opus 5 leads at 44.3 EngiScore; only 3.6% of multi-software attempts succeed.
Computer-use benchmarks score an episode by matching the final desktop state or a reference solution. That works for office apps. It fails for engineering files. A part can look right on screen and still be non-manifold, miss topological constraints, or carry loads a solver cannot run. The file dies the moment someone opens a CAE job or posts a toolpath.
Engineering-specific tests cover only one slice of the design loop. Text2CAD is mostly single-turn generation. CADWorld caps interaction at 100 steps. FEABench, BIM-Edit, and CADTestBench each sit inside one tool and one task format. Real work is a loop: pick a tool, keep parametric logic alive across a long native session, hand intermediate files to the next application, then iterate on renders, solver residuals, and design-rule reports. Without that loop, clicking a GUI and shipping a usable artifact collapse into one number.
EngiWorld drops an agent into a live Windows or Ubuntu install of professional software. A task is a six-tuple: goal, inputs, environment, action set, required deliverables, acceptance criteria. Interaction is a partially observable sequential decision process. GUI agents see 1920×1080 screenshots and click or type. CLI agents send shell commands and scripts, and may call readimg. The episode ends on DONE, FAIL, or a step budget. Scoring looks only at submitted files.
Six disjoint task types cut the loop into stages:
Coverage spans CAD, CAE, CAM, BIM, EDA, and 3D visualization on 26 platforms (FreeCAD, SolidWorks, ANSYS, Abaqus, KiCad, Revit, Blender, and others). The split is 691 CLI and 610 GUI tasks, 1,723 task-software associations, and 37 VM images: 26 single-app, 10 composite, one open workspace.
A unified domain-verifier suite reopens STEP, DXF, IFC, netlists, G-code, and related files. It checks dimensions and topology, simulation quantities, schematic connectivity, BIM entities, and cross-stage consistency. Binary tasks need every criterion to pass. Quantitative tasks receive a quality score in [0, 1] only after a feasibility gate; infeasible work is 0. EngiScore averages binary success and quantitative quality, then multiplies by 100, with equal weight per task.
Experts pull workflows from documentation, tutorials, and practice. An LLM helps standardize specs and implement checkers. Every task is executed in its target app. On the completed audit set, 96 reference artifacts all pass; 21 empty deliveries and 21 metrics-only files all fail.
Seven models run zero-shot on a stratified 300-task slice (152 CLI, 148 GUI, 73 software-by-type strata). History keeps the last 15 turns. Default budgets are 200 GUI / 100 CLI steps, raised to 300 / 150 for multi-software and open-ended work.
| Model | Overall | Single | Multi | CLI | GUI | Steps | Cost |
| Claude Opus 5 | 44.3 | 50.3 | 8.3 | 47.9 | 40.5 | 70.2 | $19.02 |
| GPT-5.6 Sol | 38.0 | 44.6 | 4.2 | 47.1 | 28.6 | 54.9 | $9.73 |
| Qwen 3.8 Max | 25.8 | 26.3 | 0.0 | 34.7 | 16.6 | 108.1 | $6.52 |
| DeepSeek V4.1 Flash | 25.5 | 30.3 | 8.3 | 48.3 | 2.0 | 116.2 | $0.81 |
| Gemini 3.7 Flash | 25.3 | 30.9 | 0.0 | 36.1 | 14.2 | 110.8 | $2.11 |
| Kimi K3 | 24.1 | 30.3 | 0.0 | 31.3 | 16.7 | 92.5 | $5.90 |
| Qwen 3.8 Flash | 16.1 | 20.0 | 4.2 | 30.8 | 1.0 | 120.5 | $0.45 |
128 of 300 tasks are zeros for every model. Across 168 multi-software attempts, 6 succeed (3.6%). Claude reaches 66.7 on image-based modeling and 25.0 on software selection. CAM is the hardest domain for every model. DeepSeek's 48.3 CLI score is the best in the table; its GUI score is 2.0, from three successful GUI tasks. Strong command-line performance does not carry over to visual operation.
On 33 quantitative tasks (231 runs), 169 scores are 0, 53 are partial, and 9 hit 1.0. The feasibility gate is hard: Qwen 3.8 Flash records RMSE 0.18097 on a ZBrush surface task and still scores 0 because the mesh has degenerate faces.
Failures cluster. 94.1% of main-eval failures are either DONE with a verifier zero or hitting the turn cap. That DONE-with-zero share is 87.0% of GPT's retained failures and 73.9% of Claude's. Turn exhaustion is 55.3% of GUI failures and 36.6% of CLI failures. Agents often treat a plausible screen as a finished job. Geometry and cross-stage consistency do not agree.
Putting the reference drawing in the prompt lifts GPT's GUI EngiScore from 15.0 to 31.7, with 15 wins and 5 losses among 60 tasks. An accessibility tree shortens episodes and raises API cost 50.7% to 147.9%, without a stable score gain. Extending Kimi's history from 10 to 15 turns drops score and raises cost 53.0%. Extra pixels are not extra engineering.
On two shared designs, the prescribed KiCad→OpenSCAD→FreeCAD→Blender path is failed by all seven models. Open-ended task 09, same geometry, is passed by Claude, Gemini, Qwen 3.8 Max, and DeepSeek via FreeCAD scripting. Several multi-software deaths are missing logs or undersized JSON, before geometry is scored.
This is a measurement paper with the widest professional-software coverage currently in the agent literature, and it grades files, not screenshots. For an engineering copilot, the bottleneck is cross-app handoff and checking the deliverable before DONE. Another single-app GUI demo will not close that gap. CLI skill does not transfer to GUI; DeepSeek is the existence proof.
The 300-task table is a diagnostic. It is not a ranking to ship against.
There is no standalone Limitations section. The main table uses 300 of 1,301 tasks. Open-ended work in that table is 4 tasks; multi-software is 24. The 3.6% figure will move with sample size. There is no human expert baseline, so 44.3 has no conversion into years of practice. Tolerance-boundary tests sit outside the completed verifier audit. LLM-assisted task writing is untested for contamination. CAD has 526 software associations and CAM 80, so 'CAM is hardest' is mixed with sample size. GUI control is screenshot plus PyAutoGUI, not native APIs. All runs are zero-shot; retrieval over manuals and domain finetuning are untested.