WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents
Jingjie Ning, Xueqi Li, Yibo Kong, Dongting Li
cs.AI, cs.LG
2026-09-23
Agents submit a full response surface graded on exhaustive CPU effects. Flash recovery at B=8 is 0.632, 0.698 with a same-point GP; code equivalences lift six-factor GP to 0.462.
Research agents edit code, run trials, and revise recipes from scores. Most benchmarks grade the final number or whether an experiment plan looks plausible. Picking a good configuration is not the same as knowing how a component change behaves when other settings are held fixed. In retrieval, one switch raises NDCG@10 by 0.160 in one background and lowers it by 0.084 in another, an interaction of -0.245. Across the catalog, 35 of 36 task conditions have a sign reversal; 24 still reverse when both sides exceed 5% of the score range. An average main effect or a single best mask does not capture that.
WhatWorkedBench defines experimental understanding as how accurately, after a measurement budget, a system predicts every conditional component effect. Each task has 4 or 6 binary options, all masks legal. The agent gets two free anchors (all-off and all-on), may buy at most B extra measurements, and must submit a complete response surface in [0,1]. The evaluator holds exhaustive native CPU outcomes. The primary target is mean absolute error on conditional effects, plus recovery relative to true effect magnitude. Pair interactions, grid MAE, and selection regret are reported too. The strict bar requires every effect error within 10% of the score range.
The catalog has 8 workflow families, 30 sources, and 36 task conditions: classification, regression, clustering, forecasting, restoration, retrieval, beat detection, and link prediction, totaling 1,248 configuration records. Numerical controls include main-effect ridge, pair-effect ridge, and Gaussian processes. A shared-estimator protocol freezes the agent's purchased observations and only changes the fitter, separating what was measured from how those points were used. Code equivalence classes turn visible rules such as "filter width is inactive when the filter is off" into measurement constraints.
On 22 four-factor sources at B=8, pair-effect ridge selects the true optimum on 15 and passes strict reconstruction on three. Thirteen exact-choice cases still miss the strict error bar. Family-macro recovery is 0.612 for pair ridge and 0.701 for an effect-variance GP.
On six matched four-factor sources, DeepSeek-V4-Flash at B=8 recovers 0.632 in the original cohort and 0.621 in the later one. Fitting a GP to those same observations raises recovery to 0.698 and 0.720. Across 18 Flash B=8 episodes, seven pick the exact optimum and one passes strict reconstruction. Filling unmeasured cells with the observation mean yields 0.287, against 0.632 from the submitted tables.
On six completed beat-detection and graph submissions at B=32, delivered family-macro recovery is 0.303; a GP on the same observations reaches 0.455, up on all six sources. On six workflows with six binary options at B=20, encoding code equivalences raises GP recovery from 0.248 to 0.462. In eight diagnostics, agents list all six registered rules before measuring and still emit tables that violate them; post-hoc projection onto equivalence classes lifts recovery from 0.338 to 0.507.
Full code versus an opaque view of masks and scores is roughly tied; shared GP even prefers the opaque observations. Seeing program structure is not the same as using it.
If auto-research benches only ask whether the score went up, optimization success is mistaken for understanding. A complete response surface graded against exhaustive native execution splits those targets. Shared estimation shows how much extra recovery a different fitter can extract from the same trials. The setting is a reproducible target for experimental design, response-surface inference, and making agents obey inactive-parameter rules. The workflows are CPU-enumerable factorials, not full GPU training recipes.
Four or six factors are what makes exhaustive references possible; real training recipes are much larger. The 108-episode core includes HTTP 429s and timeouts, scored as zero recovery. Flash and Pro cohorts from different dates differ, so interface reliability enters the denominator. Agents can recite rules and still submit tables that break them, so the code channel is not yet wired into prediction. An AI coding assistant helped implement and draft; the authors check with cold native replay and independently implemented metrics.