Delta-Matching closes the FP8 attention gap: all 7 core matmuls in FP8, loss matches BF16
songhan_mit · x · 2026-10-29
New research Delta-Matching enables all seven attention-core matmuls to run in FP8. On a 1.67B GDN/GQA model, validation CE reaches 1.4162 vs 1.4178 for BF16/FP32 and 1.8970 for naive FP8 with stale delta.
- Key idea: recompute the softmax backward's row correction δ from the operands the backward actually uses, avoiding the saved-output shortcut's mismatch caused by FP8 rounding differently in forward and backward.
- Experiments span 569M–5.29B parameters and 8K–64K context, covering QK norm/RoPE/NoPE and GDN, MLA, KDA, Mamba-2, and pure-GQA architectures.
- Notable finding: runs can track reference loss for thousands of updates while selected attention channel gains silently drift; larger models drift faster and longer runs reveal degradation short tests miss — a scaling barrier for stale-delta FP8 attention at industrial scale.
More from Infra
- Vercel Launches Agentic CI With Turborepo Task Skipping, 2-6x Faster Builds — shuding · 2026-10-29
- GitHub sees 7.4B monthly commits from AI agents, adds 4M CPU cores to cope — ai · 2026-10-29
- Seagate revenue jumps 61% to $4.2B as exabyte supply sells out through 2028 — BenBajarin · 2026-10-29
- Jensen Huang: doubling model speed doubles AI lab revenue while demand outruns capacity — rohanpaul_ai · 2026-10-29
- 400+ tool calls per turn: AIOps platform uses Monty's 1.2ms Rust sandbox to keep MCP output out of context — samuelcolvin · 2026-10-29
- Oracle's $664B backlog is real, but free cash flow is negative $23.7B — pdamodaran · 2026-10-29