FrontierChallenge: Evaluating Scientific Workflow Completion
Liangcai Su, Zhaopeng Feng, Zhuo Chen, Zhen Zhang, Xiang Lin, Ruilin Li, Handuo Zhang, Ning Wang, Kailong Wen, Yueqi Guo, Feng Xing, Yiling Guo, Chenxiong Qian, Simon Shaolei Du, Lidong Bing, Xinyu Wang
cs.AI, cs.CL, cs.SE
2026-08-26
Best setups fully complete 20 of 97 scientific workflows (20.6%). Analytical chemistry tops out at 4% pass, electrochemistry at 0%; 75.5% of failures still claim done.
Most agent benchmarks score a final answer, a single program, or work inside one discipline. A scientific handoff is a bundle: runnable analysis code, tables and figures that match, and a report that satisfies a written contract. Getting most of the way there still fails the downstream scientist if one figure or one number is missing.
FrontierChallenge asks a narrower question. Once the objective, inputs, and required outputs are fixed, can an agent finish the workflow on its own? It does not ask the agent to invent the research agenda.
The authors curated 300 end-to-end workflows and, in this release, evaluate 97 of them (74 Hard, 23 Medium). The other 203 stay held out. The 97 tasks span quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry/environment, covering 21 workflow families and tools such as ORCA, CP2K, LAMMPS, AmberTools, and PLUMED.
Each task is a self-contained package: frozen inputs, available software, an output contract, a stepwise rubric, and an executable Grader. Pass Rate counts only submissions scoring at least 99.9, i.e. full contract satisfaction. Avg. Score tracks partial progress. Semantic rubric items go to GPT-5.6 Sol as Judge, averaged over three passes. Twelve frontier models run under Codex, Claude Code, or Frontier Agent.
Full completion stays rare. Pass Rate ranges from 3.1% to 20.6%; Avg. Score from 67.5 to 87.9. The joint best Pass Rate of 20.6% (20 of 97 tasks) is shared by Codex with GPT-5.6 Sol (Avg. 87.9) and Claude Code with Grok 4.6 (Avg. 86.6). Eight configurations average above 80, and none fully delivers more than one task in five.
| Setup | Pass Rate | Medium | Hard |
| GPT-5.6 Sol + Codex | 20.6% | 39.1% | 14.9% |
| Grok 4.6 + Claude Code | 20.6% | 43.5% | 13.5% |
| GLM-5.2 + Claude Code | 3.1% | 4.3% | 2.7% |
Domain gaps are sharper. Quantum chemistry reaches 60% pass with Grok 4.6; molecular dynamics tops out at 38%. Materials characterization can average 88.1 with at most 9% pass. Analytical chemistry averages 87.6, yet only DeepSeek V4 Pro-0813 completes any task, at 4%. Electrochemistry/environment averages as high as 94.9 with 0% pass across every configuration.
Of 849 non-passing Claude Code runs among 970 trajectories, 75.5% still end with completion language. Tool errors appear in 94.2% of passing runs versus 80.7% of failures, so an error trace is not a useful failure detector. Mean runtime is 21.8 to 112.8 minutes per task; Grok 4.6 uses about 2.18M input tokens per task, Apodex 1.1 with Claude Code about 13.73M. Appendix cases include a wound-healing image assay, a tilted TLC plate, and reaction-calorimetry safety: each demands a script, tables, figures, and a report, and missing any one fails the contract.
Anyone wiring agents into lab pipelines gets a blunt measurement: high partial scores, confident wrap-ups, and tool crashes do not tell you the work was delivered. The missing pieces are contract tracking and cross-artifact checks, not another point on the average.
Rankings also hide field-specific holes. Quantum chemistry and molecular dynamics are relatively completable; analytical chemistry and electrochemistry almost never are.
The study covers 97 tasks whose official graders do not need GPUs, uses single runs, and inspects trajectories only for Claude Code. Token accounting follows provider tokenizers and cache rules, so it is not a hardware-normalized efficiency comparison. Domain slices are not a probability sample of those fields. The 99.9 cutoff only absorbs Judge averaging noise; the real blocker is the heterogeneous deliverable contract.