Information-theoretic scaling laws for diffusion models predict when memorization begins
davidad · x · 2026-10-29
New work from Luca Ambrogioni, Druv Pai, and Yi Ma gives a rigorous information-theoretic characterization of generation vs. memorization in denoising-based distribution learning from finite samples. Quantifying how much information a model learns from data predicts when memorization begins and how FID scales with training — and the mutual-information measure, despite daunting theory, turns out surprisingly efficient to evaluate.
More from Research
- Compute allocation recipe: 32 tuning runs at smallest scale, 68.7% budget at largest — karen_ullrich · 2026-10-29
- Scaling laws hold down to 4M params—if you tune small models thoroughly — karen_ullrich · 2026-10-29
- Microsoft's BioEmu2 is out: predicting protein conformational ensembles — FrankNoeBerlin · 2026-10-29
- Persona prompt pins models at ~35% on SIMMER benchmark regardless of Claude or DeepSeek — amplifiedamp · 2026-10-29
- Alfred Menezes releases gentle introduction to lattice-based crypto covering Kyber and Dilithium — matthew_d_green · 2026-10-29
- Google DeepMind, NUH Singapore Start Trusted Tester Trials of AI Co-Clinician — alan_karthi · 2026-10-29