FineWeb-30M speedrun record falls to 14.2s on one 8xH100 node, a 1.64x speedup
RobertTLange · x · 2026-11-05
Robert Tjarko Lange's team set another zero-order FineWeb-30M speedrun record: 3.5 validation loss in 14.2 seconds on a single 8xH100 node, down from the previous 22.9s record — a 1.64x speedup.
Key optimization steps:
- 22.7s: #52 rerun with identity-path phase, Cut with single roots at blocks 4 and 6, fused chain kernels, one CUDA graph per estimate
- 19.9s (×1.14): 16-row updates; each GPU natively estimates two rows in one graph with one gradient exchange and optimizer step per 16 rows; end merge became an in-run per-part fit
- 17.4s (×1.3): chain kernels moved to tensor cores (wgmma), Gluon FA3-style kernels added, weight credits in bf16, root 4 replayed only every 2nd update early in Cut
- 14.2s (×1.63): the systems tail — bit-exact pruned top-k, gathered value tables (16 MB instead of 268 MB), RoPE-rotated keys, verified bf16 recompute, rank-major ANVIL with overlapped graphs; per-Cut-row time fell from 1.76 ms to 1.07 ms
One lesson from failures: cheaper root estimates (token subsets, control variates, importance sampling) all lost — bias cost more than noise, and late-training noise hurt more than offline gradient checks predicted.
Related event: FineWeb-30M Speedrun Record Cut to 14.2 Seconds on Single 8xH100 Node(2 posts)→
More from Infra
- Apple's coreai-models repo (2.2k stars) now accepts third-party contributions — pcuenq · 2026-11-05
- oMLX lets a second Mac boost prefill up to 2.9x on big-memory Macs — pcuenq · 2026-11-05
- HydraDB: a Rust graph database that keeps state in S3 and treats query workers as disposable compute — JeremyCMorgan · 2026-11-05
- OpenSSH 10.6p2 released with a workaround for a critical vulnerability — Animux · 2026-11-05
- Microsoft rejects multibillion-dollar AI deal with UK's Nscale, FT reports — nordicinst · 2026-11-05
- Microsoft Details Running Coding AI Locally on 128GB RTX Spark Windows PCs Without Token Fees — lee_stott · 2026-11-05