PrimeScientist: Strategic Allocation of Research Effort in Autonomous Research
Xinle Yu, Fan Bai, Kaiser Sun, Hengshuo Miao, Abhay Anand, Zhongyan Luo, Kun Zhou, Zhen Wang
cs.CL, cs.AI, cs.LG
2026-09-16
Makes research-effort allocation an explicit optimization target: adaptive MCTS tightens exploration as the token budget drains, lifting reward 10.3% with 50.6% fewer attempts.
Autonomous research agents already run the full loop: propose a hypothesis, edit experimental code, run the experiment, read the result, decide what to try next. The weak spot is accounting. An agent can spin out far more directions than its budget allows, and each complete attempt burns a large chunk of tokens, so pursuing one closes the door on others. Deciding how to distribute research effort should be a core capability of these systems.
Existing systems each have a blind spot here. Karpathy's autoresearch does linear refinement: edit, run, keep or revert, and the next attempt always extends the current trajectory. Tree- and population-based systems such as AIDE or AI Scientist-v2 keep alternatives alive but navigate them with fixed search rules or LLM judgment that never consults the remaining budget. AlphaLab's strategist checks the remaining experiment count and FML-Bench switches once from greedy refinement to branching, yet the tokens spent on planning still sit outside the allocation objective. PrimeScientist makes effort allocation itself the thing being optimized.
The problem is formalized as sequential decision-making. The state is the task, the current plan set, the history, and the remaining budget. At each step the policy picks one of three actions: have a reflector (an LLM agent that drafts and revises plans) construct or revise plans, have a coding agent execute a plan, or stop. The budget is a shared pool of inference tokens, and planning and execution bill the same account. That is the load-bearing design choice: deciding what to try is itself research overhead.
Main runs use GPT-5 through the Codex CLI with a shared 1.5M-token budget per task, against linear-refinement AutoResearch and a single-run agent.
| Benchmark | PrimeScientist reward / attempts | AutoResearch reward / attempts |
| FIRE-Bench (12 AI research tasks) | 0.7738 / 13.33 | 0.7018 / 27.0 |
| AutoLab (8 systems and code tasks) | 0.4824 / 10.0 | 0.4428 / 17.5 |
| MLE-Bench (4 Kaggle tasks) | wins 2, attempts 11-15 | wins 2, attempts 21-25 |
Across the three benchmarks, 23 of 24 tasks used fewer attempts. FIRE-Bench gains 10.3% higher reward with 50.6% fewer attempts, improving on six of twelve tasks and matching on two; dropping MCQ Selection Bias, the largest single jump (1.0 vs 0.429), still leaves 0.7533 vs 0.7266, so the gain does not ride on one task. On AutoLab's accuracy-gated Smallest Game Player, AutoResearch scores 0.0 while PrimeScientist reaches 0.3224; on the other seven tasks rewards are a wash (0.5052 vs 0.5061) and everything saved is experiment count.
The ablations isolate what the allocation rule itself is worth. Swapping only the selection policy, with executor, reflector, tree, and budget accounting all fixed: 0.598 for PrimeScientist over eight tasks, against 0.563 for Greedy, 0.551 for UCT, and 0.546 for Random. Fixing the exponent instead of adapting it (GPT-5-mini): adaptive 0.537 beats fixed α=10 (0.507), uniform sampling (0.502), and α=3 (0.452), ranking first on three of six tasks and never last. Budget awareness contributes on its own.
The process-side numbers hold up too. On LLM Value Consistency, AutoResearch plateaus near 0.73 while PrimeScientist keeps climbing to about 0.89, and five of six tasks reach 90% of the competitor's best score in fewer attempts. The overhead is real: reflection, tree context, and child generation consume 38% of tokens (execution takes 62%), and one plan expansion costs 100K-300K tokens. The ledger still comes out ahead: under a flat 1M-token budget, six AutoLab tasks total 112 attempts versus 158, about 29% fewer, and mean search time drops from 42 to 32 minutes.
For anyone building agent infrastructure or automated research platforms, the transferable finding is that under a fixed inference budget, spending tokens on deciding what to try is not pure overhead. Halving attempts halves the downstream cost of training and benchmarking, and on token-billed deployments, same-budget productivity converts directly into savings. The exponent-keyed-to-remaining-budget mechanism is not tied to research either; any agent task with many candidate options and expensive validation, such as hyperparameter search, architecture search, or prompt pipelines, can adopt it, and it costs one sampling-weight formula to implement.
The honest framing: gains are double-digit percentages on AI research tasks and roughly a wash on systems optimization, a solid incremental improvement. What it establishes is the modeling choice of putting budget inside the policy, plus an evaluation discipline that reports outcomes, attempt counts, and inference cost together.
Author-acknowledged:
Concerns from reading it: there is no head-to-head against stronger tree-search baselines like AIDE or AI Scientist-v2, since the main opponent is linear AutoResearch; the evidence for beating fixed rules (UCT, Greedy) covers only an eight-task subset; on FIRE-Bench each plan node is executed twice with the higher score kept, a setting that structurally favors tree methods; and equating research quality with benchmark auto-scores deserves a discount.