Replicating 4-bit forward-pass training on one B200: 38% faster at 3.3B with identical loss

OtherRaisin3426 · reddit · 2026-11-17

After Thinking Machines said Inkling 2 Small was trained with a native 4-bit forward pass, a Redditor replicated it end to end on a single B200 (Modal, PyTorch 2.8, cuBLASLt NVFP4/FP8 via ctypes): forward in NVFP4, backward in FP8 with stochastic rounding, master weights in high precision. Results: 4-bit training ran faster (840M: 5.5 min → 4.3 min for 2,000 steps), with bigger models gaining more (18%/28%/38% for 219M/840M/3.3B), and final validation loss was essentially identical (1.69 vs 1.691). Gotchas: Blackwell's convert instruction rounds ties toward zero, breaking bit-for-bit checks, and PyTorch's attention backward isn't bit-repeatable. Full write-up, code, and a 3-hour video are linked.

Original post →

More from Infra

Infra channel →