Together's megakernel journey: built them last year, deleted them this year
simran_s_arora · x · 2026-11-17
Together team members shared their journey building the Together Compiler, via a long-form post by Dylan Lim: they helped introduce some of the first AI megakernels a year ago, and have now removed them.
- They found whole-model megakernels, while widely adopted and a useful abstraction, overcomplicate things and aren't always an ideal optimization target.
- The post does a deep dive into two contrasting ML optimization paradigms — Torch Compile vs. Megakernels — and covers early iterations of their agentic optimizer.
A substantial first-party engineering retrospective for anyone following inference stack optimization and kernel compilation.
More from Infra
- Baseten pushes DeepSeek V4.1 Flash to ~2,500 tok/s on Vera Rubin, 1.7x faster than GB300 — ying11231 · 2026-11-17
- Unsloth Dynamic v3.1 UD-IQ1_M on RX 7900 XT: 53 tok/s with 192K context — zxspectrun · 2026-11-17
- Alchemy PR adds read-your-writes: clients auto-send command receipts with every read — samgoodwin89 · 2026-11-17
- Pipeshift claims 40% faster first token than Sarvam serving Gemma 4 31B at 150B tokens/day — NirantK · 2026-11-17
- LoopFill: training-free speculative decoding speeds up looped Transformers — UIUC-CS · 2026-11-17
- Tencent's DSpine speculative decoding ships GGUFs and a llama.cpp PR: 4.5x speedup on 27B — pmttyji · 2026-11-17