Observation: Pretraining Looks Like Structured Quadratics Near End of Training
josephdviviano · x · 2026-07-31
Alex Meterez shared an observation regarding the LLM pretraining process: near the end of training, pretraining looks like a sequence of structured quadratics.
However, there are caveats to this finding: the agreement is worse during the early stages of training and when using a constant learning rate schedule compared to a cosine schedule. The author notes they are actively investigating why this happens.
Related event: Quadratic Models Can Accurately Describe LLM Pre-training Dynamics(3 posts)→
More from Research
- Goodfire's Nick Cammarata argues InceptionV1 polysemanticity wasn't caused by superposition — petrusenko_max · 2026-10-16
- LLM ideation experiment: just ask for non-obvious ideas — and Opus beats smaller models by a wide margin — arjunrajlab · 2026-10-16
- SimpleMark: multi-bit LLM watermark bounds per-token distortion with f-divergence budget — rishabh16_ · 2026-10-16
- Stanford paper: generative representation learning cuts tokenizer training compute by 7-8x — HirokatuKataoka · 2026-10-16
- Cohere Labs to present confidence estimation for reasoning models at EMNLP 2026 — Cohere_Labs · 2026-10-16
- Vals RSI Index extrapolates Google models hitting 130% autonomous AI R&D by Oct 2027 — HaydnBelfield · 2026-10-16