AutoDataBench: A Data-centric Testbed for Accelerating Auto Research
Ruifeng Yuan, Yizhi Li, Yaxin Du, Fengyu Cai, Yiqi Liu, Hou Pong Chan, Chenghua Lin, Yun Chen, Jian Yang, Bryan Dai, Pinyan Lu, Chenghao Xiao
cs.CL
2026-10-01
AutoDataBench fixes training and tests data skill alone: repair hits 81.98 vs expert 81.65, knowledge injection wins by 12.2, retrieval gains collapse on hidden evals.
Auto-research benchmarks like MLE-bench and MLAgentBench hand the agent every lever at once: training framework, hyperparameters, compute, and data. When one agent beats another, you cannot tell whether it won on tuning or on actually understanding data. Data is also the most expensive variable to validate, since every candidate choice demands a training run.
AutoDataBench isolates it. The paper defines Data Intelligence as an agent's ability to understand, manipulate, and improve training data, and sets a hard bar: data decisions must actually raise the score of the model trained on them, and the agent must change course when results contradict its expectations. If that competence holds up, the loop of models generating data, training, and evaluating gains a real foundation.
The framework nails down every non-data factor: training pipeline, auxiliary models, source datasets, and budget (one A800 GPU, a 24-hour wall clock, at most 20 training-evaluation rounds, cumulative submitted data capped at ten times the per-round maximum). Each round the LLM inspects data, writes intervention code, submits a dataset, and gets target-task feedback. One rule matters: no example-by-example manual processing, since real curation runs to hundreds of thousands of examples; bulk work must go through code and auxiliary models.
Three dimensions become three tasks, spanning three training paradigms:
Each task also hides an out-of-distribution evaluation the agent never sees, built to catch benchmaxxing: overfitting the feedback target while collapsing on unseen distributions.
Seven frontier LLMs, three runs per task:
| Task | Random baseline | Expert | Best LLM (ID) | OOD |
| Tool use (AST) | 70.45 | 81.65 | 81.98 (GPT-5.6-Sol) | 57.26–59.44, all below expert's 60.23 |
| Retrieval (nDCG@10) | 34.38 | 40.62 | 40.46 (GPT-5.6-Sol) | 26.15–30.18, all below baseline's 31.13 |
| Knowledge (Novel) | 39.10 | 48.40 | 60.60 (Kimi-3) | best Retention 50.37 (Claude-4.7) |
The three tasks tell three stories. On corrupted-data repair, LLMs sit next to the expert reference (80.82–81.98 against 81.65); that job is close to done. On knowledge injection every LLM matches or beats the expert, and Kimi-3 clears it by 12.2 points. Retrieval is the outlier: GPT-5.6-Sol nearly matches the expert on ID (40.46 vs 40.62) yet posts the worst OOD score at 26.15, below even the random baseline of 31.13, while the best OOD model (Qwen-3.7-Max, 30.18) sits mid-pack on ID. Strategies tuned against in-distribution feedback flip rank on unseen data.
Kimi-3 takes the top Overall score (60.68) on the strength of knowledge injection; GPT-5.6-Sol follows at 58.85 on tool use and retrieval ID. Across 63 runs, 61 beat their own initial datasets, with mean first-to-best gains of 8.11, 6.46, and 4.91 points; Kimi-3 climbed from 68.10 to 82.45 on tool use.
Two extensions carry the most signal. Asked to predict scores before training, GPT-5.6-Sol landed a mean absolute error of just 0.64 points on retrieval, evidence of real data-effect reasoning rather than blind trial and error, though prediction-enabled runs posted lower best scores in four of six cases. And reusing benchmark trajectories (8M tokens, upsampled tenfold to 1.6% of a 5B-token mid-training run on Qwen2.5-Coder-14B) lifted SWE-bench Multilingual from 27.67 to 33.33 and CRUXEval input prediction from 72.00 to 74.12, while MBPP gained only 0.79. The two big winners both reward inspecting evidence before acting, the same structure these trajectories have.
For benchmark builders this is a cleanly attributed slice of auto-research: with the pipeline locked, score differences can only come from data decisions. The hidden OOD evaluation is a design worth copying, cheap to add and quick to expose rankings propped up by feedback-target overfitting.
For data practitioners the results map directly onto work allocation. Structured-data cleanup is at expert level already; knowledge-style training-data construction beats hand-built rules by a wide margin; retrieval negative mining, which demands precise "hard but irrelevant" judgments, still collapses under autonomous optimization, and OOD monitoring stays necessary.
The data-engine finding may matter most: trajectories generated inside a benchmark carry training value, so the exam feeds the model after grading.
The authors state that behavioral analysis is descriptive, without isolating the causal contribution of individual operations, that first-to-best gains conflate feedback with checkpoint selection and training variance, and that the expert and baseline references carry no repeated-run uncertainty.
The prediction study deserves more skepticism: one prediction-enabled run per combination is compared against a mean of three standard runs, and knowledge-injection run variance reaches ±7.71 for Kimi-3, so the negative effect may sit within noise; the paper itself only calls the differences "lean negative." The data-engine experiment runs a single seed, model, and scale, so the +5.66 on SWE-bench Multilingual has no confidence interval. The expert references also differ in construction across tasks, so they are not strictly one bar.