Science sandboxes measure the scientific capability of AI agents
Arya S. Rao, Rodrigo I. Castro, Sager J. Gosai, Kenneth B. Hsu, Yasha Ektefaie, Shantanu Singh, Sangeeta N. Bhatia, Steven K. Reilly, Ryan Tewhey, Eric S. Lander, Pardis C. Sabeti
q-bio.QM, cs.AI
2026-08-31
Science sandboxes score both metric and reasoning. Claude one-shot MPRA libraries hit median r=0.774 vs human 0.763, then could not name hidden invented rules.
Most science-agent scores reward the last number: a high-activity sequence, a correct structure, a right answer. Experimental science asks something else. Evidence should change the hypothesis, and the next experiment should be more informative than the last. A rising score does not mean the system was understood.
The Broad Institute, with Jackson Lab and Yale, introduce science sandboxes. An agent queries a sealed oracle, writes a lab notebook, and is judged on both the metric and the reasoning. Oracles come in three grades: wet physical assays, damp models trained on real data, and dry rules invented by the designers. The same loop can test use of known biology, or induction when the rule is unfamiliar.
A sandbox has specimens, assays, and an oracle. This paper instantiates two biological tasks.
In MPRAbox the agent picks 50,000 sequences of 200 bp regulatory DNA from a space of size 4 to the 200, to train a downstream sequence-to-activity model. Labels come from Malinois, a CNN trained on 776,474 experimental MPRA measurements (Pearson r about 0.88 across three cell types). A fresh model of the same architecture is trained from scratch and scored on 14 hidden test sets. The agent sees only the 14 Pearson correlations and their mean. Human baselines are 14 expert library strategies, five sampling seeds each.
CodonBox hides a genetic code. Alphabet size j is 4, 6, or 8; codon length k is 2, 3, or 4; residues are hydrophobic or polar; folds follow Dill's 2-D lattice; fitness is the count of non-adjacent H-H contacts in the best fold. The agent submits one sequence per round for 500 rounds and sees a single scalar.
Agents: Claude Opus 4.7 in Claude Code, GPT-5.5 in Codex, Gemini 3.5 Flash in the Gemini CLI. Five independent one-shot replicates each. Thirty-round play, dry oracles, and CodonBox were mostly Claude.
Without human strategy scores, Claude's median r was 0.774. All five runs met or beat the best human strategy mean of 0.763. Gemini sat at 0.680 and GPT at 0.655; neither beat the best human library. Notebooks diverged: Claude sampled real genomic regulatory DNA and upweighted rare classes in all five runs; GPT built synthetic motif perturbations in all five; Gemini mixed the two.
After seeing human strategy summaries (not sequences), GPT and Gemini shifted toward genomic DNA. Claude moved from 0.774 to 0.781, GPT from 0.655 to 0.760, Gemini from 0.680 to 0.751. Nobody copied the winning human recipe.
Claude's four 30-round trajectories all beat the strongest human strategy. Quantitative gains over one-shot were modest. The notebooks show hypothesis revision: random DNA was expected near zero and scored high, so the agent decided composition signal is cheap, and the real job is teaching regulatory grammar.
Swap in 14 invented rules and score and rule discovery come apart. The English-word cipher never prompted a letter-encoding hypothesis in any framing. The Fibonacci-position rule was read as dinucleotide composition. In CodonBox, repeated-character strings hit the maximum score within 10 rounds every time. With j=4 the agent recovered codon length, but not the full table at k=4. With j=6 and k=3 it never found codons at all and invented a run-length theory. The hardest world, j=6 and k=4, 1,296 codons, yielded groups of four; the silent position and the interaction rule did not.
The paper cuts into science-agent leaderboards: beating Malinois correlation is not doing science. If an agent is choosing real experiments, familiar priors make it look expert; unfamiliar rules let it burn the budget on a score-improving wrong theory. BroadBox is meant to open more sandboxes to the community.
This is an evaluation frame, not a new model. The damp oracle is Malinois, so the agent is optimizing libraries that train well on Malinois.
The authors say wet assays are expensive, slow, and noisy; damp oracles inherit model bias; dry rules give exact ground truth and drop physical complexity. Long-horizon, dry, and CodonBox runs are almost all Claude, so they do not speak for every frontier agent. Agents were told not to reverse-engineer oracle weights; logs say they complied, which is an instruction, not a sandbox. Rule discovery is scored by reading stated hypotheses. The text never totals how many of the 14 dry rules were actually named.