Together Deleted Its Megakernels: AI Agents Tune Inference 28% Faster on B200
togethercompute · x · 2026-11-17
Together AI's engineering retrospective explains the road to Together Compiler, which uses agents to rewrite and fuse GPU kernels for up to 2.6x throughput. Key points: last year's megakernels hit record speeds but were impractical (one model, one batch size, one customer after months); surprisingly, CUDA Graphs plus PDL on default Torch Inductor kernels recovers 98% of megakernel performance with zero custom kernels; agent-driven "multi-kernels" then beat handwritten megakernels by 28% (2,118 vs 1,649 tokens/s on Llama 3.2 1B, B200). By Dylan Lim, Reyna Abhyankar, and Simran Arora.
Related event: Together AI Drops Megakernels for New Compiler with 28% Speedup on B200(2 posts)→
More from Infra
- One Chase software bug rippled across US banking: a case for resilient infrastructure — AryHHAry · 2026-11-17
- United Airlines hit by 'widespread system outage' days after national ground stop — shaunralston · 2026-11-17
- Cloudflare tops 10 million developers on its platform, more than doubling in a year — Beth_Kindig · 2026-11-17
- AWS adds memory-aware agentic retrieval to Bedrock: long-term memory plus per-user ACL isolation — AWS ML Blog · 2026-11-17
- NVIDIA open-sources cuPotential, 3x faster molecular dynamics on Blackwell GPUs — AllThingsApx · 2026-11-17
- Flights, banks and Microsoft hit by mass outages; officials deny cyberattack — AaronBergman18 · 2026-11-17