TwinLab-Bench: Evaluating AI Scientists in Agentic Twins of Real Labs
Yang Wan, Linchao Zhu
cs.AI, cs.CL, cs.LG
2026-11-19
TwinLab-Bench tests experiment design on 52 twins of real labs. Claude Fable 5.2 averages 45.0 out of 100; with a zero budget, Opus 5.5 and DeepSeek-flash pass none.
Science-agent benchmarks mostly stop at two checks: can the model run a workflow it was handed, and can it pick the next measurement from a menu someone else wrote. Designing the experiment is the next step, and the reading has to agree with a real laboratory. A real lab is slow and expensive. Simulators emit numbers from fixed rules. Replays only answer conditions that were already run. A language model cast as the lab drifts. When DeepSeek-flash replays requests from real labs, task text alone makes 41% of readings deviate from the record and 31% substantially disagree on a repeat. Given the raw files, those rates are still 14% and 8%.
TwinLab-Bench, from Zhejiang University, has 52 tasks: 20 in the physical sciences, 20 in the life sciences, and 12 in engineering, with more than 100 action types. The tasks were adapted from about 375 sources. Forty-nine sit on real records; three are grounded syntheses. One task asks whether odd Shapiro steps are missing, against 2,149 raw files.
Each record is cut at a date or a lab state. Data, protocols, and interim conclusions from before the cut go to the actor agent. Later findings are the hidden truth, visible only to that task's TwinLab agent and to the judge. The actor writes experiments in natural language. New readings come only by asking the environment, and each request spends budget, counted in lab days or wells. On most tasks the budget is one to three times the expert route. Web search blocks the source site and the names of the benchmark and the tasks.
The TwinLab agent is the lab. It looks through the raw files and returns the logged value when a comparable condition is on record. Otherwise it runs a calculator, a deterministic program built from that lab's data: resampled measurements, a fitted physical model, or an instrument export template, returned in the instrument's own format. A quantity that neither the record nor the literature constrains is marked as a simulation limit, costs nothing, and is not an experimental result. Calculators extend the record only locally. A sample that has been used stays used.
The traps are already in the record. Scarce material is not replaced. A result that looks decisive can have another cause. On some tasks the correct report is that the budget cannot settle the question. The rubric is frozen before the final runs and scaled from 0 to 100. Conclusion items check the answer. Verification items check that this run produced the evidence. Any core item that misses full credit zeros the task, and full credit is rechecked twice. Replaying the lab's expert route in the twin averages 97.8.
Thirteen models act as the actor, in 17 reasoning setups. GPT models run in Codex; the others run in Claude Code. The TwinLab agent and the judge are DeepSeek-flash on every run. With DeepSeek-flash as the actor, the environment and the judge add about 55 USD across the 52 tasks. Fable 5.2's actor runs cost 424 USD; Opus 5.5's cost 141 USD.
| Setup | Metric | Result |
| Claude Fable 5.2 | Mean score on 52 tasks | 45.0 |
| Claude Opus 5.5 | Mean score on 52 tasks | 40.2 |
| GPT-6.1-sol | Mean score on 52 tasks | 31.7 |
| Lowest configuration | Mean score on 52 tasks | 5.4 |
| Expert-route replay | Mean score | 97.8 |
| Zero budget, 33 tasks | Passes by Opus 5.5 and DeepSeek-flash | 0 |
| Task text only (12-task pilot) | Readings off the record | 41% |
| TwinLab agent (same pilot) | Off-record / inconsistent | 2% / 4% |
Score rises with the log of actor cost. On 19 of 21 tasks that drew both a correct and a wrong report, every correct report scores higher, by 0.46 of the maximum on average. With a normal budget, at least one model passes 28 of those 33 tasks.
Across 7,808 requests in 831 runs, 76% of readings come from the original record or from a calculator calibrated on that lab. Of 189 decisive experiments on the expert routes, 78% use conditions the record already contains. Calculators rebuilt without held-out points miss by a median of 0.74% when interpolating and 2.47% when extrapolating. Replacing the extrapolation with another plausible choice flips the pass verdict in 2 of 12 matched runs; the median score change is 0. In successful runs, more than a third of requests are experiments the expert route never makes.
Closed-book, tools off, Opus 5.5 fully recalls no source conclusion and derives the answer without data on only 2 tasks. Fable 5.2 recalls one. With the handoff text frozen and only the lab's true state changed, the conclusion follows the new state in 8 of 10 states that allow a verdict.
On tasks that need an experiment, Opus 5.5 scores 0 on 16 runs. Twelve of those are decision errors, seven of them accepting a mechanism with no discriminating test. An oyster sugar assay reads about 30% low; the model clears the pipettes with a water check and never tests the 1 µL transfer, which delivers only about 70% of the sample. Decisions the plan states in full later score 87% of their maximum; decisions the plan never mentions still score 62%. Of 11 fully stated decisions the run still loses, 9 are carried out as planned and the points are lost while reading results or writing the report. Of 21 failed runs, only 8 spend at least 90% of the budget, and doubling it saves 2.
The expert route scores 97.8, so the tasks can be solved. The 45.0 average is low because the experimental decisions are wrong. When the decisive data are absent from the handoff, a zero budget yields no passes: the points come from measurements made in the run. Grading the written plan would mis-rank the work. Fully stated decisions still die in the reading or the report, and decisions left out of the plan are often executed anyway. Failed runs usually have budget left, and more budget rarely flips them. Clearing a core item takes a measurement that separates one mechanism from another.
The extrapolation is load-bearing on some tasks. On radiocarbon pretreatment, the correct conclusion depends on how large an extrapolated residue is, and one model changes its advice for low-yield samples when that choice changes. The swap covers 6 tasks. The held-out check covers 3. In 7,808 replies, one states a hidden mechanism without being asked, and the judge does not flag it. That run scores 0. Another reply describes an observation the calculator does not model. Told the lab is simulated and may be exploited, Opus 5.5 is blocked on all 9 attempts over 4 tasks, and 13 real flaws it names do not change scores. That probe is small.
The judge is itself a language model. Opus 5.5 agrees on the pass verdict in 19 of 20 runs, but a core item on a wording boundary still drops from full credit to zero and wipes the task. Most actors run once. Three full passes of DeepSeek-flash have a standard deviation of 3.3 points. The tasks were built with Claude Code and Opus 5.5, which is also a model under test. Nothing checks whether the wording favors that family. The actor never touches an instrument.