Claude-written megakernels for 8 open models hit up to 1.77x speedup over vLLM/SGLang
joao_gante · x · 2026-11-19
A team had Claude write a separate megakernel for each of 8 open models (8B to 594B): one GPU kernel that runs the entire decode loop, declaring the end of one-size-fits-all inference engines.
- Up to 1.77x faster than vLLM / SGLang
- 2.05x with lossless model-specific tricks
The approach replaces generic inference engines with per-model full-loop kernels — an aggressive direction in inference-stack optimization.
More from Infra
- US blocks Chinese firms from renting Nvidia chips abroad while Microsoft nears $500M OpenAI deal for ByteDance — kevinnbass · 2026-11-19
- vikhyatk: we are overinvesting in GPU compilers, underinvesting in CPU compilers — mohbibi_ · 2026-11-19
- KVARK Flash: a proxy that caps DeepSeek's thinking per request, 30-50% fewer tokens — Ill_Contribution_717 · 2026-11-19
- Polymarket launches Anthropic IPO perps, ANT trades near $104 ahead of Nasdaq debut — Polymarket · 2026-11-19
- R9700 at 160W for llama.cpp: ~20% pp loss, near-silent fans — LippyBumblebutt · 2026-11-19
- Baseten runs DeepSeek v4.1 Flash at ~2,500 tok/s on Vera Rubin, 1.7x faster than GB300 — baseten · 2026-11-19