Training AI Scientists to Replicate Research
Damon Falck, Samer Sabri, Anja Surina, Thom Foster, Anya Sims, Sam Devlin, Dylan Rogers, Tantum Collins, Kaloyan Aleksiev, Louis Kirsch, Edward Hughes
cs.LG, cs.AI
2026-08-13
Inherent Labs post-trains Qwen3.6-27B on 310 paper-replication tasks with a rubric-based judge, beating Claude Opus 4.8 by 6% and GPT-5.5 by 8% on held-out tasks.
Replication is how science checks itself, and it happens to be good training for AI scientists: the task is underspecified by construction (a paper is a lossy compression of the research behind it), it demands hypothesis-driven exploration, and there is no crisp reward to hill-climb on, so existing agent harnesses do not naturally apply. Frontier agents struggle with exactly this shape of problem. Inherent Labs turns "replicate one figure" into a scalable training task and uses it to post-train a 27B scientist agent.
Three components:
| Comparison | Result |
| In-distribution ML tasks beating Claude Opus 4.8 | 73% |
| Held-out AI-for-science tasks beating both baselines | 60% |
| Average test score vs Claude Opus 4.8 | +6% |
| Average test score vs GPT-5.5 (Codex) | +8% |
Twenty-four rounds of automated prompt optimization on Codex did not meaningfully close the gap. Human experts preferred Faraday in 29 of 41 head-to-head rollouts. The qualitative gap is the telling part: baselines hard-code expected outputs or over-simplify, while Faraday implements the mechanism under test. On the Darwin-Gödel Machine task, a baseline hard-coded the discovered agent while Faraday implemented the evolutionary search itself; on GNoME, baselines reported a single seed per point while Faraday reported uncertainty across multiple seeds. All agents degrade on more recent papers; NLP and LLM papers are hardest and classical ML easiest.
This is a solid data point for the "post-training teaches taste" route: a small general model, aligned to the task, beats other labs' frontier models on an underspecified research workload. The recipe of rubric judge, multi-sample averaging, and turn-level credit assignment transfers to any long-horizon agent task where reward is not verifiable. If the 310 tasks and the generation pipeline are released, they become a community asset in the SWE-bench mold.
The authors call this early stage: Faraday fails on several tasks where the original research was rigorous, and the judge was never validated on "innovation" tasks. The human-preference study only covers rollouts where the rubric judge already scored Faraday ahead, so it cannot establish that humans prefer Faraday on average. Training covers only 60-minute, one-seventh-GPU horizons; full-scale replication is untested. The test split has 68 tasks, so a +6% average margin is not large at this task count. The judge is Codex while one baseline is also Codex, a familial relationship worth watching. All evaluation is in-house and awaits independent replication.