New scaling law paper quantifies data synergy across 52 open-weight LLMs, extends Chinchilla
LesterMackey · x · 2026-10-09
'Domain-Aware Scaling Laws Uncover Data Synergy' (Hamidieh, Mackey, Alvarez-Melis) extends Chinchilla by adding pairwise domain interactions to the scaling law, fitted observationally across 52 open-weight LLMs. It formalizes 'data synergy' — domain mixtures whose combined contribution exceeds or falls short of the sum of parts (e.g. code boosts math reasoning; some mixtures interfere) — estimating both direct domain-to-benchmark and second-order domain-domain synergy. The framework outperforms domain-agnostic scaling laws in predictive accuracy, and validation runs on predicted optimal and anti-optimal mixtures confirm the estimates correctly predict performance rankings.
More from Research
- EMNLP paper: diffusion LMs internally encode latent denoising timestep representations — gsarti_ · 2026-10-23
- KU Leuven and VIB open two Group Leader professorships in AI for Biology, start 2027 — jpillowtime · 2026-10-23
- Subspace Tuning curbs LLM catastrophic forgetting by freezing principal singular directions — scottniekum · 2026-10-23
- Perceptic launches PercepticOS, an AI workspace for 'scientific superintelligence' — nathanbenaich · 2026-10-23
- EPFL's Bosselut to talk long context as test-time learning at ELLIS seminar — andre_t_martins · 2026-10-23
- DRATS: Adaptive Task Sampling Boosts Worst-Task Performance in Multi-Task RL — Cohere · 2026-10-23