Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents
Jingjie Ning, Shanshan Zhong, Xiaochuan Li, Ji Zeng
cs.MA, cs.AI, cs.SE
2026-09-07
DCP audits research agents on sealed utility, no-lineage recovery, and feedback. Two DeepSeek audits: 0/96 recoveries (bound 0.0468) and 30/30 vs 0/30 feedback.
AI research agents already return programs, recipes, and experimental plans with a score attached. A sealed test can show that the artifact is useful. It cannot show that the score required the run's experimental diary, rather than the starter kit, public pages, and the model's prior.
Carnegie Mellon turns that gap into three executable questions about one numerical outcome. Did the artifact beat a baseline on sealed evaluation? Can a matched agent, given the same background and the same observed web bytes but none of the target run's history, still hit the score? From a frozen checkpoint, does truthful experimental feedback raise the chance of reaching the target relative to a specified neutral channel? A high score is the start of the audit, not the certificate.
The Discovery Certification Protocol (DCP) names five objects: background K fixed before the run (model, task rules, starter code, tools), initial observations E0 that do not depend on the run's actions, research history L created during the target run and consumed by later actions, final artifact A chosen by a pre-registered rule, and a machine-checkable recovery predicate P.
Gate 1 tests utility. On sealed evaluation, the lower confidence bound on A minus baseline must clear a registered minimum useful gain δmin. Both artifacts must be valid, and the baseline must sit below the recovery region.
Gate 2 tests recovery. Fresh matched agents receive the same K, complete E0, model, tools, and budget, plus every web byte the target run actually saw (Wobs). They do not receive L. Any valid artifact scoring at least x−ε is a qualified recovery witness and vetoes Core on the spot. With zero recoveries in n independent episodes, a finite-sample upper bound on the episode recovery probability pB must fall at or below a registered ρ, and positive controls, opportunity budget, and information access must all pass. Every valid method counts: a different implementation, a transfer, or a recombination of known parts.
Gate 3 is optional and tests feedback. Paired branches start from a registered checkpoint c. One arm gets truthful measurements from its own actions; the other gets messages that keep timing, schema, and length but do not point to the correct next action. Evidence additionally requires that, on independent null tasks whose answers are already in E0, the neutral channel sits inside a registered equivalence band ±δsham against a second non-informative channel, and that the feedback-effect lower bound clears δevidence+δsham.
Two decisions follow. DCP Core is useful improvement, adequate controls, zero recoveries, and a passing bound. DCP Evidence adds a calibrated checkpoint-conditional feedback effect. Weak controls, broken interfaces, or failed registration yield audit incomplete. A deterministic, LLM-free verifier replays every decision from frozen evidence bundles.
Two complete three-gate audits, different models and tasks, same decisions.
| Task | Model | Main / baseline | Recovery line | Gate 2 | Decision |
| SQLite event service | DeepSeek-v4-flash | 0.8855 / 0 | 0.8805 | 0/96, best 0.6734 | Core+Evidence |
| Virtual catalyst | DeepSeek-v4-pro | 1.0000 / 0.5990 | 0.9500 | 0/96, best 0.8146 | Core+Evidence |
| Device calibration | Flash | 1.0000 / 0.3667 | 0.9500 | 0/80, best 0.7133 | Core |
| Multidimensional knapsack | Flash | 0.9349 / 0.9140 | 0.9329 | recovered at 0.9363 | Core vetoed |
The SQLite task picks four partial indexes among 16 query families, of which four carry hidden high traffic. Score is the traffic-weighted cut in SQLite virtual-machine steps versus a no-secondary-index baseline. The target run read two sqlite.org pages through a recording gateway and scored 0.8855 on the sealed workload, an 88.55% reduction. Challengers received every captured web byte and peaked at 0.6734. The catalyst task has five controls with eight legal levels each (32,768 recipes). The agent commits an anchor, receives a 36-well one-factor plate, then submits one static recipe. Score 1.0 means all five hidden optima and optimal behavior over 256 sealed conditions. Challengers peaked at 0.8146.
Both complete audits recorded 0/96 recoveries, pupper=0.0468 under ρ=0.05, and 45/45 positive controls with recall lower bound 0.8889 against a 0.8 floor. Each Gate 3 study produced 30/30 truthful recoveries and 0/30 neutral recoveries, with a paired-binary 99% interval of [0.6379, 1.0]. Separate 60-pair null studies gave a contrast of 0 and interval [-0.0950, 0.0950], inside the registered ±0.17 band. The verifier returned Evidence in both scopes.
Knapsack is the constructive recovery case: the target beat its baseline by about 0.021, and one matched episode returned two legal solutions above the 0.9329 line. A low-sample affine run failed control adequacy and stayed incomplete. Full SQLite evidence used 507 model sessions and $61.17; catalyst used 435 sessions and $56.40. Decisions now replay offline.
Leaderboards ask whether an agent can raise a score. DCP asks whether the score has sealed utility, whether a matched agent can recover it from the starting packet, and whether the feedback channel is doing work. For groups shipping AI Scientist-style systems, that is a shared evidence language: artifact hashes, information boundaries, episode budgets, recovery bounds, feedback intervals. A reviewer can rerun the verifier instead of trusting the write-up.
It is a measurement protocol, not a new search algorithm. The tasks are controlled software optimization and virtual experimental control, not open-ended science. Formal issuance still needs independent provenance checks and registry countersignatures; every bundle in the paper has issued=false.
The audit tasks were designed with known information boundaries. Hidden high-traffic SQLite families and hidden catalyst optima exist to exercise Core, recovered, and incomplete decisions. They do not show that an agent discovered new science.
Gate 2 challengers do not receive new task scores or scientific measurements, only non-directional messages. That tests recovery without the experimental diary, not whether a second agent running a full truthful campaign would independently find the same outcome. Zero hits in 96 episodes still leave an upper bound of 0.0468, not zero. The paper separately reports a 0.1619 upper bound on neutral recovery at 30 episodes, keeping average feedback benefit and residual neutral recovery apart.
Both complete audits used DeepSeek-v4 models, with Claude CLI as the execution layer. A new model or budget needs a new registration. Web access is byte-for-byte replay of captured pages, not open search. Historical priority is recorded as not assessed.