SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline
hsu_byron · x · 2026-11-12
LMSYS published a technical blog on building a full-layer (MoE + attention) megakernel from scratch in CUDA C++/PTX for Muse Spark Open, lifting decoding from 517 to 1,109 tok/s on 8×GB300 — one kernel launch per layer, with launches per decode step dropping from 1,450 to 97.
Key techniques:
- Prefetching weights into shared memory and L2 so every memory load overlaps other work (extreme latency hiding).
- Warming the instruction cache — Nvidia's instruction cache makes big kernels stall fetching their own code; they tuned instruction and data caches at all levels.
- Twelve warps with different roles; some warps of the next op start before the previous op finishes, redefining GPU utilization.
- Using data itself as a readiness signal for cross-GPU communication; fusing the draft model made speculative decoding 2.6× faster.
Includes measured timelines, CUDA/PTX examples, and tradeoff analysis; next steps are larger batches and toward an "operating system for the GPU."
Related event: LMSYS Megakernel Boosts Muse Spark Decoding to 1,109 tok/s(2 posts)→
More from Infra
- Big AI conflates scale with innovation as voters reject data center ballot measures — DavidLinthicum · 2026-11-12
- Musk says SpaceX targets 1TW/year from Earth, rising to 1PW/year from the Moon — ns123abc · 2026-11-12
- Where does Claude's cost really go? Running build tools may quietly add up — IgorBrigadir · 2026-11-12
- Int8 AVX512 VNNI quantization cuts CPU inference to 34ms vs 90ms fp32—2.6x faster — wavefnx · 2026-11-12
- H100 rental rates run 54% above straight-line depreciation, a16z finds — a16z · 2026-11-12
- Apple trains small offline nets to route RAG queries, trading upfront cost for faster retrieval — petrusenko_max · 2026-11-12