vLLM cuts Flint's kernels from 22 to 7 per MoE layer with custom fusion
vllm_project · x · 2026-10-22
The vLLM project collaborated with Recursive to make Flint fast from day one: custom kernels and aggressive operator fusion reduce kernel launches from 22 to just 7 per MoE layer at batch 1. The shared diagram details the fused path — attention side: Add+RMSNorm, QKV GEMV (fusing Q/K norm, RoPE, KV-cache write), decode attention with sinks and split-KV merge, and Output GEMV; MoE side: Add+RMSNorm with router top-6, Gate/up GEMV with shared expert and SiLU, Down GEMV with weighted combine. Runs NVFP4 at up to 8 requests; dense layers 0-1 replace the MoE row with a norm and two GEMVs.
More from Infra
- SPCX earnings call highlights 'computer per watt' metric as AI value shifts down the stack — mobileraj · 2026-10-22
- Eureka Machine runs on Mac inference so quietly the author's MacBook barely spins its fans — cong_ml · 2026-10-22
- Compute demand is so hot it's becoming a month-to-month auction, analysts say long-term contracts are dying — BenBajarin · 2026-10-22
- SGLang ran early RL training on TPUs with Miles and TorchTitan, with promising results — ying11231 · 2026-10-22
- CXMT's market cap tops all mid-size Asian fabs combined, half of SK Hynix — pstAsiatech · 2026-10-22
- SemiAnalysis: DeepSeek's open-sourced DeepInfer lacks key inference-serving components — cHHillee · 2026-10-22