Together AI Drops Its Own Megakernels, Ships Agent-Tuned Compiler 28% Faster on B200
On November 17, the Together AI team (Dylan Lim, Reyna Abhyankar, Simran Arora, and others) published a long-form retrospective on two years of inference optimization research. A year ago, they were among the first to introduce AI megakernels (whole-model fused kernels) to the world; this year, they deliberately removed their own megakernels in favor of a new Together Compiler (Polymorphic Inference), which runs 28% faster than a hand-written megakernel on the B200, with throughput gains of up to 2.6x.
Confirmed
- The team's conclusion: megakernels are too complex to serve as an optimization target and are not an ideal abstraction; Torch Compile's surprisingly strong performance also helped drive the shift to a compiler approach.
- The core claim of the new Together Compiler: the optimal compiler is not always a megakernel compiler—the key is identifying which groups of operators should be coupled and tuning them individually.
- Agentic systems are well suited to navigating this enlarged search space, as AI agents can automatically tune inference for the model, chip, and traffic.
- Measured results: 28% faster than a hand-written megakernel on the B200; MiniMax M3.1 Flash improved 1.88x to 1299 tokens/s; overall throughput up to 2.6x.
Why it matters
- This is a rare case of a leading inference team publicly acknowledging the limitations of its signature technical approach (megakernels) and rebuilding from scratch, offering valuable lessons for the broader AI compiler community.
- Using AI agents for compiler search and tuning represents a new direction in automating inference optimization, potentially reshaping the "hand-written kernels vs. automatic compilation" competition.
2026-11-17 ~ 2026-11-17 · 10 related posts
- Episode 1: Together AI Launches TKCC, an Agent-Driven Continuously Compiling Inference Optimizer(2026-11-05, 28 posts)
- Episode 2: Together AI launches agentic continual compiler TKCC for faster inference(2026-11-05, 4 posts)
- Episode 3: Together Compiler boosts Kimi K3 throughput up to 2.7x(2026-11-10, 2 posts)
- Episode 4: Together AI Drops Its Own Megakernels, Ships Agent-Tuned Compiler 28% Faster on B200(2026-11-17, 10 posts)
Primary sources
- Together Deleted Its Megakernels: AI Agents Tune Inference 28% Faster on B200 — togethercompute ·
- [source] Together Deleted Its Megakernels: AI Agents Tune Inference 28% Faster on B200 — togethercompute · 2026-11-17
- Together explains why it dropped megakernels for agent-tuned inference compilation — togethercompute · 2026-11-17
- Torch Compile underrated: CUDA Graphs + PDL hits 98% of megakernel speed on Llama 1B — simran_s_arora · 2026-11-17
- Together Compiler benchmarks: 1.88x on MiniMax M3.1 Flash, gains across Kimi K3 and GLM 5.3 — simran_s_arora · 2026-11-17
- Together deletes its megakernels: tuned Torch Inductor matches hand-written kernels — HazyResearch · 2026-11-17
- Researcher Praises Together's Torch Compile vs Megakernel Tradeoff Study — simran_s_arora · 2026-11-17
- Last Year We Built Megakernels. This Year We Deleted Them — simran_s_arora · 2026-11-17
3 near-duplicate retellings: simran_s_arora · simran_s_arora · simran_s_arora