Observation: Pretraining Looks Like Structured Quadratics Near End of Training
josephdviviano · x · 2026-07-31
Alex Meterez shared an observation regarding the LLM pretraining process: near the end of training, pretraining looks like a sequence of structured quadratics.
However, there are caveats to this finding: the agreement is worse during the early stages of training and when using a constant learning rate schedule compared to a cosine schedule. The author notes they are actively investigating why this happens.
Related event: Quadratic Models Can Accurately Describe LLM Pre-training Dynamics(3 posts)→
More from Research
- Research: Adapting TimesFM 2.5 for Price Volatility Distribution Forecasting — const_reborn · 2026-07-31
- Guidelabs: Interpretability Must Be Built Into Training, Not Added Afterward — jefrankle · 2026-07-31
- Goodfire Traces LLM Neurons to Steer Models Away from Endorsing Drunk Driving — leedsharkey · 2026-07-31
- AI Autonomously Tunes Quantum Devices Across Physical Platforms with 75 Diagrams — bravo_abad · 2026-07-31
- ICLR New Policy: Reviewers Must Disclose AI Tool Usage and Report Original Assessments — TuhinChakr · 2026-07-31
- Arena Introduces AutoEval: Minute-Level Ratings via Reward Models — arena · 2026-07-31