Stanford paper proposes rewriting scaling law: optimal synthetic data share rises to 32% with compute
stanfordnlp · x · 2026-10-31
A paper by Nicholas Roberts, Ahmed Ahmed, Suhas Kotha, Percy Liang et al. proposes a "rewriting scaling law" that predicts loss from model size, training tokens, the fraction of rewrite tokens, and epochs over natural data — while charging the compute spent generating rewrites. Key findings: in the compute-limited setting, the optimal rewrite fraction grows with compute, from under 10% at 3×10^18 FLOPs to 32% at 10^22 FLOPs; extrapolating 33x in FLOPs to train a 22B-parameter model yielded a final loss within 1.1% of prediction; and that model beats its rewriter in BPB on several web domains using one-tenth of the rewriter's training compute. A data-limited setting with a same-size rewriter is analyzed as well.
Related event: Stanford Researchers Propose Scaling Laws for Synthetic Data Rewriting(3 posts)→
More from Infra
- Clover-35B-A3B Quantized to 29.4 GB Runs at ~42 tok/s on a 64 GB Mac — viveky259 · 2026-10-31
- Liquid Agent brings fully local LFM3 agent to Linux with one-command install — JosephJacks_ · 2026-10-31
- A fully local office AI stack: MiniCPM, Qwen3, LFM2.5 and Whisper — HowDevelop · 2026-10-31
- a16z: GPU rental prices up 106% in 3 months while token prices fall 58% — a16z · 2026-10-31
- vLLM nightly adds PLE disk offload, with a telling [Qwen4Exp] tag on the PR — rmhubbert · 2026-10-31
- Solo AI studio: per-minute GPU rentals rendered a 328-frame Blender shot for ~$1 — r0ck3t23 · 2026-10-31