New scaling law paper says repetition can beat paraphrasing for some pretraining regimes
burny_tech · x · 2026-07-29
A new paper proposes compute-data scaling laws to bridge the gap between compute-optimal and data-optimal pretraining.
- Classical scaling laws assume unlimited fresh data, but the paper argues pretraining is increasingly constrained by data availability.
- The authors introduce a token-effectiveness function η that measures how valuable derived tokens are relative to fresh ones.
- They fit the framework on Dolma-3 across model sizes from 14M to 600M parameters, studying two expansion strategies:
- multi-epoch repetition
- paraphrasing
- Key findings:
- token effectiveness is not constant; it depends on model size, tokens-per-parameter ratio, and derived-data volume;
- effectiveness saturates as the corpus expands;
- adding compute in place of data has diminishing returns as models and data get larger;
- classical compute-optimal allocation is suboptimal across most practical settings.
- The framework divides training into compute-bound, data-bound, and model-bound regimes and suggests repetition can beat paraphrasing in some model/corpus-size ranges.
More from Infra
- Reflection AI open-sources Beam under Apache 2.0 with 40+ ecosystem partners — zhyncs42 · 2026-10-16
- NVIDIA launches RTX Spark for local AI on Windows PCs — ostrisai · 2026-10-16
- SkyRL ships Day-0 RL support for Beam with +62% LoRA decode throughput in vLLM — vllm_project · 2026-10-16
- DFlash2 delivers up to 3.03x decode-only speedup for Beam across five benchmarks — vllm_project · 2026-10-16
- vLLM's Beam serving kernels cut GB200 forward time from 8.15 to 6.47 ms at TP8 batch 128 — vllm_project · 2026-10-16
- Reflection's 501B MoE Beam hits 463 tok/s on 8x GB200 with NVFP4 at day 0 — zhyncs42 · 2026-10-16