Quantifying Pretraining Variance: Floating-Point Order Effects Nearly as Large as Data Seeds
AndrewLampinen · x · 2026-08-19
A new blog post quantifies sources of variance in LLM pretraining. Beyond data and initialization seeds, floating-point arithmetic order (e.g., from sharding differences) also affects results, with effects nearly as large as data seeds.
More from Research
- Stanford Team Wins Databricks Grounded Reasoning Cup With 63.3% Accuracy Via End-to-End Agent Optimization — jefrankle · 2026-08-19
- Grok 4.6 lands #2 on new DiligenceBench, statistically tied with Claude Opus 5 — karinanguyen · 2026-08-19
- GPT Doubles Formalized Erdős Solutions Over a Weekend, Proving Autoformalization Works — AlexKontorovich · 2026-08-19
- Building a Benchmark for Decomposition Attacks for Agents at AI Safety Night — chhaviyadav_ · 2026-08-19
- Researchers found 3 confirmed CVEs in Chromium, VirtualBox, QEMU for $609 — thedealdirector · 2026-08-19
- ACL 2026 Paper: LLMs Stick to Old Knowledge Despite New Information — mdredze · 2026-08-19