LMSYS megakernel pushes decoding to 1,109 tok/s on 8× GB300, launches cut 1450→97
ying11231 · x · 2026-11-12
LMSYS published a technical post on Muse Spark Open Ultrafast: a from-scratch CUDA C++/PTX megakernel raising decoding from 517 to 1,109 tokens/s on eight GB300s. Each layer (attention + MoE) runs as one kernel, cutting launches per decode step from 1,450 to 97. Techniques include prefetching weights into shared memory/L2, warming the instruction cache, using data as a readiness signal for cross-GPU comms, and fusing the draft model for 2.6× faster speculative drafting. Includes measured timelines, CUDA/PTX examples, and tradeoff analysis.
Related event: LMSYS Megakernel Boosts Muse Spark Decoding to 1,109 tok/s(2 posts)→
More from Infra
- Big AI conflates scale with innovation as voters reject data center ballot measures — DavidLinthicum · 2026-11-12
- Musk says SpaceX targets 1TW/year from Earth, rising to 1PW/year from the Moon — ns123abc · 2026-11-12
- Where does Claude's cost really go? Running build tools may quietly add up — IgorBrigadir · 2026-11-12
- Int8 AVX512 VNNI quantization cuts CPU inference to 34ms vs 90ms fp32—2.6x faster — wavefnx · 2026-11-12
- H100 rental rates run 54% above straight-line depreciation, a16z finds — a16z · 2026-11-12
- Apple trains small offline nets to route RAG queries, trading upfront cost for faster retrieval — petrusenko_max · 2026-11-12