Together explains why it dropped megakernels for agent-tuned inference compilation
togethercompute · x · 2026-11-17
Together AI published a retrospective on its inference optimization journey: a year ago the team helped introduce some of the first whole-model megakernels, but has concluded they overcomplicate things and aren't an ideal optimization target — hence "this year we deleted them."
Key points:
- Deep dive into two contrasting ML optimization paradigms: Torch Compile vs. megakernels;
- Torch Compile showed surprisingly strong results, challenging the team's megakernel assumptions;
- These experiments led to Together Compiler, announced two weeks ago: agents that tune inference to your model, chip, and traffic, delivering up to 2.6x more throughput;
- The team shares the experiments and lessons shaping its next-generation inference optimization system.
Related event: Together AI Drops Megakernels for New Compiler with 28% Speedup on B200(2 posts)→
More from Infra
- Together Compiler: AI agents rewrite GPU kernels, beating hand-written megakernels by 28% on B200 — simran_s_arora · 2026-11-17
- Together Compiler benchmarks: 1.88x on MiniMax M3.1 Flash, gains across Kimi K3 and GLM 5.3 — simran_s_arora · 2026-11-17
- DeepSeek v4.1 Flash hits ~2,500 tok/s on Vera Rubin in Baseten low-latency tests — baseten · 2026-11-17
- ARK Newsletter Flags Jensen Huang's Resignation as Major Test for Nvidia — downingARK · 2026-11-17
- Tracing Nvidia's circular financing: $170B in sales, only ~$35B unsettled — tengyanAI · 2026-11-17
- One Chase software bug rippled across US banking: a case for resilient infrastructure — AryHHAry · 2026-11-17