BF16 loses resolution as mean logit grows, breaking softmax probabilities
micahgoldblum · x · 2026-11-03
Part 7 of Goldblum's thread: the answer is finite precision. In BF16, the gap between representable values grows with magnitude, so as the mean logit grows, resolution of logit differences is lost — and those differences determine softmax probabilities, explaining why mean-logit drift breaks training.
Related event: AdamW-Induced Logit Drift Fails under BF16 Finite Precision(2 posts)→
More from Infra
- Volcano Engine's console AI agent analyzes your cloud bill to explain negative balance — vista8 · 2026-11-04
- 8 free GitHub repos with 792K stars that take your AI off Big Tech's servers — alex_verem · 2026-11-04
- OpenClaw cut its ~$1M/month CI bill by over 95% with adaptive agentic testing — vincent_koc · 2026-11-04
- LlamaIndex stacks 17 small optimizations for ~10x faster OCRia-1 token generation on a single B200 — llama_index · 2026-11-04
- phonon: WebGPU browser TTS hits 26x real-time on iPhone with a 44MB model, open sourced — blixt · 2026-11-04
- GlobalWafers at full capacity across all wafer sizes, ASP hikes expected from 2027 — BenBajarin · 2026-11-04