Quadratic Models Accurately Predict LLM Pretraining Loss, Study Shows
ShamKakade6 · x · 2026-07-30
A recent study reveals that the simplest model in optimization theory, the quadratic, accurately describes LLM pretraining. By linearizing a 150M parameter model at various training checkpoints, researchers found that the local Taylor expansion tracks the true loss effectively for up to 10% of the training budget. The paper is predicted to become highly influential in the AI community.
Related event: Quadratic Models Can Accurately Describe LLM Pre-training Dynamics(3 posts)→
More from Research
- UPenn lab shows 'critique and revise' prompting doubles Claude's research ideation output — arjunrajlab · 2026-10-16
- LLM ideation experiment: just ask for non-obvious ideas — and Opus beats smaller models by a wide margin — arjunrajlab · 2026-10-16
- What prompt gets the most interesting ideas out of LLMs? A researcher's ongoing experiment — arjunrajlab · 2026-10-16
- SimpleMark: multi-bit LLM watermark bounds per-token distortion with f-divergence budget — rishabh16_ · 2026-10-16
- Stanford paper: generative representation learning cuts tokenizer training compute by 7-8x — HirokatuKataoka · 2026-10-16
- Cohere Labs to present confidence estimation for reasoning models at EMNLP 2026 — Cohere_Labs · 2026-10-16