vLLM day-0 support for Reflection Beam: 1,769 tok/s on 4x B300 NVFP4
vllm_project · x · 2026-10-16
vLLM ships day-0 serving support for Reflection's Beam, a sparse MoE (501B total / 23B active), via the vllm/vllm-openai:reflection-beam Docker image plus an official recipe.
- Architecture: 52 layers mixing 512-token sliding-window attention with NoPE full-attention layers, 8 of 256 experts routed per token (dense MLPs in layers 0-2), 1M context, Harmony-format reasoning/tool calling, optional DFlash2 speculative drafter.
- Measured throughput (chat test, concurrency 16): 4x B300 NVFP4 TP4 hit 1,769 output tok/s; 8x B300 FP8 TP8 hit 1,300. Different GPU counts and precisions, BF16 KV, variable output lengths.
- Memory: NVFP4 weights 294GB (4x B200/B300); FP8 503GB (8x H200); BF16 1,003GB (8x B200/B300). NVFP4 quantizes only routed experts — attention, dense MLPs, router, embeddings and LM head stay BF16; on Hopper, FP4 experts run through Marlin with BF16 activations.
- SkyRL also adds day-0 RL support (LoRA, BF16 full fine-tuning, MXFP8 RL), reporting +62% LoRA decode throughput from right-sizing expert adapters in vLLM: 276 → 447 tok/s/GPU at 32 concurrent requests.
More from Infra
- llama.cpp v0.7.0: one-liner serves Qwen3.8-Flash with MTP spec decoding on 128GB Macs — ggerganov · 2026-10-17
- NVIDIA: Beam-1.0 Throughput on GB300 NVL72 Already Up From 70K to ~90K TPS per GPU — NVIDIAAI · 2026-10-17
- 4x3090 rig vs DGX Spark: the local LLM hosting trade-offs worth knowing — Dapper_Chance_2484 · 2026-10-17
- SGLang Summit 2026 lineup unveiled; Decagon to detail RL pipelines scaled to ~1T-param models — BanghuaZ · 2026-10-17
- Local LLM test: trading 21K context for higher quants cut coding time from 8h to 6h — Boricua-vet · 2026-10-17
- TeenyWork prototype turns browser PoW challenges into Bitcoin mining to price agent access — stonebrigade · 2026-10-17