vLLM details Muse Spark Open cross-hardware stack: 960 tokens/sec on 8 TPU v7 chips
vllm_project · x · 2026-10-16
The vLLM team detailed cross-hardware serving optimizations for Muse Spark Open (part 4/6).
- NVIDIA: FlashInfer + vLLM work includes fused norm/gating/residual kernels, TEP pooler/unpooler sharding with fused MNNVL collectives, and an FP32-norm-to-TF32-router path
- AMD: MI355X supports FP8 and MXFP4 with TP/PP on ROCm
- TPU: initial TorchTPU support; with DSpark (acceptance length = 6), megakernels reach over 650 tokens/sec on 4 TPU v7 chips and 960 tokens/sec on 8 chips; without speculation, over 270 and 350
- A companion post (3/6) describes Inferact's DSpark speculator: block diffusion proposes up to 7 draft tokens per pass, confidence-based adaptive verification, sliding-window attention keeps drafting lightweight; supports CUDA and ROCm
Related event: vLLM and Inferact Optimize Muse Spark Open Serving(2 posts)→
More from Infra
- DOT-MoE: converting dense models into sparse MoE via optimal transport — burkov · 2026-10-16
- Stanford CS336: A 7B model needs 14GB for weights but 84GB to train — stanfordnlp · 2026-10-16
- New NVFP4 trick: compressing block scales saves 45-48% per expert matrix, bit-exact — TheZachMueller · 2026-10-16
- AWS extends Claude Sonnet 3/3.5/3.6 EOL on Bedrock to November 13, 2026 — repligate · 2026-10-16
- After Apple and Meta CEOs tout Mac Silicon inference, a $10B distributed network idea resurfaces — gajesh · 2026-10-16
- NVIDIA's 74B Nemotron 3.5 Bolt, top US open-weight model, now runs locally on Mac — SumitGup · 2026-10-16