Together Deleted Its Megakernels: AI Agents Tune Inference 28% Faster on B200

togethercompute · x · 2026-11-17

Together AI's engineering retrospective explains the road to Together Compiler, which uses agents to rewrite and fuse GPU kernels for up to 2.6x throughput. Key points: last year's megakernels hit record speeds but were impractical (one model, one batch size, one customer after months); surprisingly, CUDA Graphs plus PDL on default Torch Inductor kernels recovers 98% of megakernel performance with zero custom kernels; agent-driven "multi-kernels" then beat handwritten megakernels by 28% (2,118 vs 1,649 tokens/s on Llama 3.2 1B, B200). By Dylan Lim, Reyna Abhyankar, and Simran Arora.

Related event: Together AI Drops Megakernels for New Compiler with 28% Speedup on B200(2 posts)→

Original post →

More from Infra

Infra channel →