Together Compiler: AI agents rewrite GPU kernels, beating hand-written megakernels by 28% on B200
simran_s_arora · x · 2026-11-17
The Together team (Dylan Lim, Reyna Abhyankar, Simran Arora) publishes a two-year retrospective on inference optimization, introducing Together Compiler (Polymorphic Inference):
- Uses AI agents to rewrite and fuse GPU kernels in a model's forward pass, hitting 2,118 tokens/s on Llama 3.2 1B on B200 vs 1,649 tokens/s for hand-written megakernels (+28%).
- Their earlier megakernels set speed records but were hard to productionize — months of effort yielded one model, one batch size, one customer.
- Key finding: default Torch Inductor kernels plus CUDA Graphs and PDL recover 98% of megakernel performance; multi-kernels and agent loops push further.
More from Infra
- Together Compiler benchmarks: 1.88x on MiniMax M3.1 Flash, gains across Kimi K3 and GLM 5.3 — simran_s_arora · 2026-11-17
- DeepSeek v4.1 Flash hits ~2,500 tok/s on Vera Rubin in Baseten low-latency tests — baseten · 2026-11-17
- ARK Newsletter Flags Jensen Huang's Resignation as Major Test for Nvidia — downingARK · 2026-11-17
- Tracing Nvidia's circular financing: $170B in sales, only ~$35B unsettled — tengyanAI · 2026-11-17
- One Chase software bug rippled across US banking: a case for resilient infrastructure — AryHHAry · 2026-11-17
- Together Deleted Its Megakernels: AI Agents Tune Inference 28% Faster on B200 — togethercompute · 2026-11-17