Stanford 22B run hits pre-registered loss forecast within 1.1%, scaling laws for synthetic data rewrites
stanfordnlp · x · 2026-10-31
Stanford NLP amplifies a research thread on how much pretraining compute should go to synthetically rewriting your data. The team fit scaling laws to answer two questions: how much compute rewriting deserves, and whether a model trained on rewrites can beat the rewriting model. A 22B hero run at 33× their fit range landed within 1.1% of the pre-registered loss forecast, validating the scaling law fit.
Related event: Scaling Laws Answer How Much Compute to Spend on Synthetic Data Rewriting(4 posts)→
More from Research
- David Duvenaud: gradient-free 'poor-man's RL' on scaffolds works but is far slower — DavidDuvenaud · 2026-10-31
- BlackboxNLP keynote: How LLMs read, from multi-token words to text in images — davidbau · 2026-10-31
- Baselabs Digs Into RL's Resurging Compute Question: Learn a Value Function or Spend on More Rollouts? — baseten · 2026-10-31
- OpenAI Claims Proof That L = BPL, Hinting at AI-Generated Formal Mathematics — chaumian · 2026-10-31
- Georgia Tech's EPSBench: LLM agents still trail analyst consensus on earnings forecasts — GeorgiaTech · 2026-10-31
- A power-law task difficulty model explains why equal capability gains produce uneven eval improvements — teortaxesTex · 2026-10-31