Reflection's 501B MoE Beam hits 463 tok/s on 8x GB200 with NVFP4 at day 0
zhyncs42 · x · 2026-10-16
TokenSpeed partnered with Reflection to serve Beam from day 0 on NVIDIA GB200. Beam is a 501B-parameter MoE (23B active) with a 1M-token context window.
Recommended setup: the NVFP4 checkpoint on two GB200 compute trays, combining fused routing/norm kernels, TensorRT-LLM NVFP4 MoE kernels, and a DFlash draft model for speculative decoding. Measured on multi-turn agentic coding workloads:
- 463 tok/s single-user, 223 tok/s per user at 8 concurrent users
- NVFP4 weights use 36.5 GiB per GPU vs 119.1 GiB for BF16, leaving more room for KV cache
- Splitting Beam's 43 sliding-window layers into smaller cache groups fits 3.07x more tokens in the same KV pool
- NVFP4 MoE kernels run 2.2–2.9x faster than BF16 at typical batch sizes; the 1.93B DFlash drafter (15-token proposals) boosts decode 1.71x
AMD MI355X is also supported, with an open deployment guide.
More from Infra
- Meta's open Muse Spark needs 493GB of memory — far beyond any consumer laptop — TejasKumar_ · 2026-10-16
- CoreWeave plunges 20% as credit markets price >60% default odds; is this dot-com all over again? — kimmonismus · 2026-10-16
- MiniCPM-V 4.7 35B-A3B hits 17+ tokens/s on iPad with ~3GB memory — techNmak · 2026-10-16
- China rolls out 'token factories' and a 2026-28 token economy action plan — nordicinst · 2026-10-16
- Vercel Sandbox Drives go GA: attach TBs of data in sub-second, fork instantly — cramforce · 2026-10-16
- NVIDIA launches RTX Spark for local AI on Windows PCs — ostrisai · 2026-10-16