DataPrep-Bench: Benchmarking LLMs as Training Data Preparators
Hao Liang, Qifeng Cai, Yibo Lin, Jianzhuo Du, Qifeng Xia, Sizhe Qiu, Linzhuang Sun, Meiyi Qiang, Zhaoyang Han, Xiaochen Ma, Bohan Zeng, Ruichuan An, Conghui He, Wentao Zhang
cs.LG, cs.CL
2026-05-19
First unified benchmark for LLM data prep, scored by downstream performance. Finds synthetic domain data often hurts SFT; releases DAS, the top cross-model data-quality scorer.
Training data caps what a large language model can do, yet "using an LLM as a data engineer" has never been measurable. Groups pushing data-synthesis and quality-scoring methods each run on different sources, base models, and downstream evals, so conclusions do not transfer. Everyone claims their method wins, but nobody sits the same exam.
The paper splits the job into two capabilities: data construction (turning raw material into supervised data) and data quality evaluation (predicting, before any training, how much a candidate dataset will actually help). "Quality" here means downstream training utility, not how clean the prose reads. The two tracks had been studied in isolation; DataPrep-Bench is the first to judge both under one shared downstream-grounded protocol.
Six domains: General Text, Math, Science, Medical, Finance, Law. Raw sources are domain books and long-form material (Wikibooks, FreeBookCentre, Open Textbook Library), converted to Markdown with MinerU.
Construction track: every method gets the same sources and produces a supervised set; it is scored by fine-tuning a base model on that output jointly with Dolly-15k as a shared instruction-following corpus, then testing on held-out domain benchmarks. Bases are Qwen2.5-7B and Llama-3.1-8B. Forcing every method to compete on top of Dolly makes each one prove its data actually adds value, rather than just producing a bigger pile.
Quality-evaluation track: each domain has a candidate pool of 8 to 14 public SFT corpora mixing in- and out-of-domain data (OpenR1-Math-220k, UltraChat, Tulu 3, MegaScience, UltraMedical). A scoring function assigns each set a scalar; the function's quality is the Pearson correlation between its scores and true downstream performance. Higher correlation means it can tell in advance which data is worth training on.
The authors ship two methods. Data-Construction-Skill is a skill-guided agent that pulls "what counts as valid supervision" out of the prompt and into a reusable skill layer (task instructions, output schema, filtering rules, validation tools); the agent extracts knowledge propositions and generates concept, reasoning, and case-based QA. DAS (Distributional Alignment Score) sits on the evaluation side and comes from domain-adaptation theory: align the training distribution with the target and downstream holds up. It encodes a dataset with a fixed encoder and measures the MMD distance (maximum mean discrepancy, a measure of how far apart two distributions are) to a high-quality domain-proxy dataset; smaller distance means a higher score.
The counterintuitive headline: mixing synthetic domain data on top of Dolly-15k often hurts. The plain Dolly-only baseline beats a host of fancier synthesis methods in several domains.
| Setting (Qwen2.5-7B, Math) | GSM8K | AIME24 | Avg |
| Dolly-only | 69.9 | 0.0 | 23.6 |
| Best agent (Qwen3.5-Plus) | 72.7 | 3.3 | 25.2 |
| DataFlow-Skill | — | — | 19.0 |
The methods also sort by domain cleanly. DataFlow-style pipelines win in structured domains like Finance and Law; pure agents lead in reasoning-heavy Math and Medical; the skill-guided Data-Construction-Skill is strongest in knowledge-extraction-dense domains. On Llama-3.1-8B Finance it lifts the average from Dolly-only's 15.1 to 34.2, nearly 20 points absolute (run with Claude Opus 4.6).
On the evaluation side, DAS is the only scorer clearing r > 0.70 in Math, Science, and Medical at once.
| Domain (3-model avg) | DAS | Qurating-Expertise |
| Math | 0.86 | 0.69 |
| Science | 0.72 | 0.72 |
| Medical | 0.71 | 0.68 |
By contrast, diversity metrics like BERTVendi reach only 0.28 on Math and stay below 0.30 in most domains. The cost is speed: DAS takes 306 seconds per dataset versus 0.03 to 85 seconds for quality-style metrics. But predicting what is worth training beats training it and finding out it did not help.
For anyone building pretraining or post-training data pipelines, this paper hands over a reusable exam bench and two ready methods. The practical takeaway is blunt: do not assume synthetic data helps. Run something like DAS before mixing it in, and you may skip a lot of dead training. The skill-driven route, which moves data specs out of prompts into a versionable, auditable skill layer, is the part that matters for productionizing this.
Keep expectations calibrated, though. This is methodology and measurement work, not a leaderboard model. Its value is putting a previously everyone-for-themselves field under one ruler.
The authors concede several hard ones. Candidate pools are small: Finance, Law, and Medical have only 8 to 10 candidates each, weakening statistical power. The skill-guided builder loses to the strongest agents on Medical and Law, and trails even the Dolly-only baseline on Science under both base models. DAS depends on the base model and drops below significance on Science for one model.
A further concern is Finance and Law. The authors call them "difficult for every metric we tested," yet DAS still posts the top correlation there, so "best" may only mean tallest among the short. DAS also leans on choosing a good domain-proxy dataset (Infinity-Instruct, ODA-Math-460k, and the like); whether the same correlations hold in domains without a ready-made proxy is unverified.