ASI-Bench: At the Dawn of Artificial Superintelligence
Junwei Zhou, Zhen Sun, Binyu Li, Jiangyu Zhou, Yuexi Pan, Hengyu Wang, Honghe Ren, Xiaohan Jia, Xueyang Zhou, Xiaoyu Cao, Yongchao Chen, Yuanning Feng, Junhao Wu, Cheng Zhang, Sijia Chen, Haoyu Xue, Chengsong You, Huan Wang, Koutian Wu, Peigan Gao, Jiakun Wu, Wenzhe Li, Ergan Shang, Qingyuan Zheng, Jingjing Zhou, Ruixuan Jia, Yan Xu, Hongrui Zhang, Xiao-Han Ma, Zhengxiang Cheng, Yuexing Hao, Liting Mai, Xianglin Ji, Wenjun Zhang, Zhuofan Chen, Yixiao Huang, Chi Wang, Wenyue Hua, Yilun Hao, Yuantao Zhai, Ziyan Zhao, Jingyan Xie
cs.AI
2026-08-18
ASI-Bench, built by 40+ experts over 31,000 hours, tests 18 agent-model pairs on 60 project-level research tasks under a four-level guidance gradient: removing the detailed procedure alone costs 21.8 points on average, while removing the method name costs only 2.5 more.
Existing benchmarks test one of two things: knowledge questions with known answers (HLE, GPQA), or execution tasks where humans have already specified the goal and method (Terminal-Bench, MLE-Bench, PaperBench). Neither answers whether AI can carry a research project end to end, because the solution path was supplied by people.
The approach here is direct: take one research task, hold the objective, data, required outputs, and scoring fixed, and progressively withdraw methodological information. How fast scores fall measures how deep the dependence on human guidance runs.
ASI-Bench was built by 40+ researchers across 13 institutions led by Tsinghua, at a cost of over 31,000 human hours. From more than 1,300 candidate research ideas, five review rounds, 1,100+ review assignments, and 1,500+ sandbox runs produced 60 project-level tasks across 11 domains: mathematics, physics, chemistry, biology, astronomy, materials, earth science, medicine, CS, robotics, and electrical engineering. Each task carries a scientific objective, data, an executable environment, and verifiable artifacts. Completing them means the full research loop: understanding, method selection, implementation, experimentation, failure diagnosis, iteration, validation. The 60 tasks total more than 2,600 interaction turns, 2,400 execution steps, and 35+ hours of agent execution.
The guidance gradient has four levels, illustrated by a 2D nonlinear dynamics task. B1 provides the governing PDE, numerical formulation, and solver procedure. B2 names only the class of PDE and suitable numerical approaches. B3 gives only spatio-temporal observations and the objective. B4 adds plausible but task-irrelevant information to B3.
This design separates following a procedure, operationalizing a specified method, and conducting research independently.
Across 18 Agent×Model configurations (6 harnesses × 14 models, including Codex, Claude Code, Kimi Code, and OpenHands), macro-averaged over three independent runs:
| Setting | Average | Drop |
| B1 (full method + procedure) | 50.91 | -21.82 |
| B2 (method name only) | 29.10 | -2.48 |
| B3 (method self-determined) | 26.62 | +0.36 |
| B4 (B3 + distractors) | 26.99 | - |
Two structural findings matter more than the totals.
First, the bottleneck is not method selection but operationalization. B1 to B2 costs 21.82 points; B2 to B3 only 2.48. Telling the system which method to use barely helps; expanding that method into a numerical formulation and solver steps is what the 21 points buy. Distractors in B4 change almost nothing. The paper names this the method operationalization bottleneck.
Second, the harness reshapes what a model can express. MiMo V2.5 Pro goes from 16.17 under MiMo Code to 23.25 under Claude Code; Kimi K2.7 rises from 19.72 to 27.34. Kimi K3, by contrast, moves less than a point between the two. Scientific capability is a product of model and harness, not of the backbone alone.
On cost: the strongest configuration, Codex + GPT-5.6 Ultra, reaches 51.60 at B3, the only system above 50, at roughly $1,550 per run. GPT-5.6 xhigh costs about $684 for 40.86, matching Claude Opus 5 with Claude Code at 40.70 despite a $2,728 price tag: a quarter of the cost for the same score. Raising reasoning effort from xhigh to ultra lifts B3 from 40.86 to 51.60. B2 turns out to be the most expensive setting, 6.91M tokens and 49.7 minutes per task, 59% more tokens than B1: being told the direction but not the procedure leaves agents constrained while still reconstructing missing steps.
For agent engineers the leverage point is clear: gains come from turning methods into executable research workflows, not from better method selection or robustness to distractors. For evaluation, a single aggregate research score is meaningless; the B1-B3 gradient is the signal, and model comparisons are invalid without reporting the harness. The benchmark is open with a contribution channel for new tasks.
The authors acknowledge that 60 tasks are a starting point. Structural concerns: reviewing, task authoring, and reference answers all come from the building team, and Chinese-built models are well represented among the 18 configurations, so task selection and review perspective may carry systematic bias. Scores are weighted-scorer outputs; the paper does not validate alignment with independent expert judgment of research quality. Standard deviations of a few points (B3 std ±3.65 for the strongest configuration) make rankings in the 50-point band fragile. All evaluation ran in sandboxes without external tool access; real research settings with literature search and databases could shift the B2/B3 difficulty relationship.