CausalMix: Causal Inference Picks Fine-Tuning Data Mixtures Without Trial and Error
The Batch (Andrew Ng) · rss · 2026-10-16
Researchers from Tsinghua, Ant Group, and Renmin University propose CausalMix, which disentangles data mixture from data quality when optimizing fine-tuning datasets. Using 512 100K-example subsets of Tulu 3 with quality metrics from OpenDataArena-Tool, they train tree ensembles (LightGBM, CausalForestDML) to estimate each mixture's marginal gain, then average the top-100 of 100K sampled mixtures. Fine-tuning Qwen2.5-7B on 800K examples, CausalMix averages 62.28% vs DMO's 60.35%, without re-running trials at 7B scale, and transfers to Qwen3-4B and code/math data (66.66% vs 63.47%). Caveat: it underperforms DMO (25.04% vs 26.49%) on a held-out Tulu 3 subset, so cross-dataset generalization is uncertain.
More from Research
- Stanford and Databricks use AI agents to reproduce medical findings and test analytical robustness — ChengleiSi · 2026-10-23
- EMNLP paper: diffusion LMs internally encode latent denoising timestep representations — gsarti_ · 2026-10-23
- KU Leuven and VIB open two Group Leader professorships in AI for Biology, start 2027 — jpillowtime · 2026-10-23
- Subspace Tuning curbs LLM catastrophic forgetting by freezing principal singular directions — scottniekum · 2026-10-23
- Perceptic launches PercepticOS, an AI workspace for 'scientific superintelligence' — nathanbenaich · 2026-10-23
- EPFL's Bosselut to talk long context as test-time learning at ELLIS seminar — andre_t_martins · 2026-10-23