Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecasts
Jingjie Ning, Xueqi Li, Yibo Kong, Dongting Li
cs.AI
2026-09-29
Across 336 prospective states, research-agent explanations made repeated forecasts more consistent (drift down 64.5%) but showed no confirmed accuracy gain for predicting outcomes.
Automated research systems in the AI Scientist mold propose an experiment and attach a rationale for why the change should work. Benchmarks score the achieved result; the rationale itself goes unexamined. It might be packaging, or it might contain real insight about the experiment. This paper makes that testable: insert an agent's explanation into a forecaster's context and check whether forecasts of the experimental outcome improve. The design draws on simulatability research, which scores an explanation by how well it helps an observer predict model behavior, and on LLM scientific forecasting. The target here is the numerical effect change of an executed external experiment, and every forecast is frozen before any outcome is visible.
Paired forecasts. Each state combines public information (parent validation metrics, training summaries) with a planned intervention. A fixed forecaster, DeepSeek V4 Pro, makes two independent calls in each of three contexts: D, description only; M, description plus the current state's explanation (matched card); S, description plus another state's explanation (donor card, assigned under a frozen mapping). The contrast RD - RM measures gain over description; RS - RM measures advantage over a mismatched explanation. The donor is the load-bearing control: if a borrowed explanation does the same job, the gain has nothing to do with fitting the current case. Two independent calls separate repeat drift (the gap between the two forecasts) from true error, so more consistent output cannot pass for more accurate prediction.
Three prospective studies cover 336 states: 120 in a controlled fast-ML environment, 72 on Tox21 molecular assay endpoints, and 144 on OpenML tabular tasks (LightGBM under one of six interventions). The forecast target is the held-out child-minus-parent effect. Five checks wrap the design: commitment (a six-slot prediction card), delivery of content into the prompt, gain against trivial priors, donor alignment, and sensitivity to known signals. All 3,708 prospective slots froze before outcomes opened.
All three preregistered decisions came back negative or inconclusive:
| Gate | Outcome |
| v5 frozen decision | inconclusive |
| Tox21 ROC AUC interval-score harm test | unmet: M-D = +.0026, 95% CI [-.0104, +.0174]; gate required ≥ .005 with a positive lower bound |
| OpenML joint rule (formation, point equivalence, repeatability) | unmet: 20/144 complete cards against a frozen .50 threshold |
Point accuracy never showed a confirmed gain. Tox21 point MAE was .02056 (description), .02041 (matched), and .02023 (donor), effectively flat, and a Flash replay moved matched MAE the wrong way, from .01823 to .02020. Direction accuracy is mostly a sign prior: 109 of 111 explicit directions in v5 predicted improvement, scoring 58.6% against 57.7% for always predicting improvement; OpenML came in at 63.19% versus a 62.5% constant-positive baseline. The only contrast in v5 with a fully positive interval was log-loss point MAE for donor versus matched (+.0508), but interval scores worsened in the same setting and the frozen decision stayed inconclusive.
What explanations did change is consistency:
| Setting | Metric | Description | Matched | Donor |
| Tox21 Pro | repeat drift | .01004 | .00357 (-64.5%) | .00411 (-59.1%) |
| Tox21 Flash replay | repeat drift | .00704 | .00278 (-60.6%) | .00288 (-59.2%) |
| Positive control, 40 states | point MAE | 6.391 | 3.795 (true mechanism) | 8.151 (false mechanism) |
Add a structured explanation and the two forecasts move closer together, without moving closer to the truth. The positive controls show the assay has sensitivity: a researcher-authored true-mechanism note cut point MAE by 2.60 percentage points (CI [2.314, 2.880]), a false mechanism note raised error to 8.151, and an exact-signal calibration won 144/144 states with Spearman .99996. The forecaster can use strong information; agent-written explanations, at the tested resolutions, do not carry it.
Two side findings on calibration: full-card assignment on OpenML widened nominal 80% intervals by 21% while coverage fell from 51.4% to 49.3%, and the six-slot extraction pipeline produced only 66 nonempty cards out of 144, while a direct-text Flash replay that passed all 144 raw notes into the prompt still found no matched gain. The three prospective studies cost about USD 18.5 combined.
This adds a missing scoring dimension to research-agent benchmarks: beyond the result, what is the rationale worth. The protocol is cheap and reusable, frozen paired forecasts with a donor control, and the negative finding is the useful part. Explanations from current automated research systems do not yet carry measurable predictive information about the experiments they justify; anyone ranking experiment proposals with LLM forecasts should demand a demonstrated predictive gain before building a cost-utility rule on top. This is a measurement paper with a negative result, not a new model.
The authors state that the alignment test is limited by donor resolution: Tox21 and OpenML donors only swap seeds within a cell, the median realized seed gap is .01362 ROC AUC, and 27 of 36 Tox21 pairs share an effect sign, so matched and donor explanations should genuinely be similar; on OpenML only 13 of 72 donor pairs had content on both sides and one pair had quantitative claims on both, leaving the alignment contrast badly underpowered. A single model family (DeepSeek V4 Pro for the prospective studies, Flash for replays) makes every call. The numbers-only component result is exploratory and unadjusted for multiplicity, by the authors' own account. Two more concerns from the read: effect sizes sit near a noise floor, with point MAE around .02 against natural seed-to-seed variation of .014, and the positive control is researcher-written, so it validates the assay without rescuing agent explanations. Post-outcome follow-ups used already-known outcomes under separately frozen plans.