Rewriting scaling law study: optimal synthetic data share rises from 10% to 32% with compute
heghbalz · x · 2026-10-26
A paper by Princeton, Stanford, and Jane Street researchers (incl. Percy Liang) proposes a "rewriting scaling law" that predicts loss from model size, training tokens, the fraction of rewritten tokens, and epochs over natural data, while charging the compute spent generating rewrites.
Key findings:
- Compute-limited setting: the estimated optimal rewrite fraction grows with compute, from under 10% at 3×10^18 FLOPs to 32% at 10^22 FLOPs.
- Extrapolating the scaling law 33× in FLOPs to train a 22B-parameter model yields final loss within 1.1% of the prediction; that model beats its rewriter on BPB across several web domains using one-tenth the rewriter's training compute.
- Data-limited setting: with a same-size rewriter pretrained on a fixed seed set, the optimal rewrite fraction also rises with compute.
- A model trained on its own rewrites can beat the best natural-data-only run up to 3B scale.
Takeaway: synthetic rewriting choices are predictable via scaling laws, and training need not plateau at the rewriter's capability level.
More from Infra
- Sail equivalence proof and riscv-coq cross-checks uncover real RISC-V spec bugs — xennygrimmato_ · 2026-10-27
- xAI's Colossus 1 hits ~14.6 exaflops FP64, ~5x Top500 leader — cis_female · 2026-10-27
- Lambda runs 19 B200 nodes in a 16-node power budget, boosting throughput 24% — TheZachMueller · 2026-10-27
- Podcast: Micron's HBM roadmap, Meta's AI infra, and NVIDIA's supply chain lock-in — BenBajarin · 2026-10-27
- SGLang MegaKernel + PD Pushes the TCO-Latency Frontier on GB300, SemiAnalysis Benchmarks Show — AccBalanced · 2026-10-27
- Scaling self-hosted Convex: compiling TypeScript Convex functions to WASM — leventov · 2026-10-27