Together AI ships TKCC: 1200+ TPS on DeepSeek v4.1 Flash with 2 nodes in 24 hours
simran_s_arora · x · 2026-11-05
Together AI has released Together Compiler (TKCC), billed as the first continual inference optimization system designed for the era of "polymorphic" inference, built from scratch by Reyna Abhyankar, who left her PhD 16 months ago to work on this.
Key points:
- Problem: AI workloads change fast—new models, diverse architectures, shifting traffic. Existing systems trade off measurement throughput vs. fidelity: agent-based kernel optimization on offline benchmarks gives distorted timings for data-dependent primitives like MoE, while end-to-end engine optimization suffers from slow engine launches.
- Approach: A Tiered Persistent Engine decouples engine launch overhead from forward-pass computation, plus a cost model that picks parallelism configurations to seed the optimization. TKCC runs as a continual agent hill-climbing loop.
- Results: Workload-specialized optimizations reach 1100+ TPS on MiniMax M3.1 Flash and 1200+ TPS on DeepSeek v4.1 Flash on the InferenceX benchmark—24 hours and just 2 nodes per model.
- How it got here: The team started by endlessly hacking Inductor (lowering to PTX, modifying, recompiling) to "automatically make megakernels," and scrapped many iterations before landing on a system that scales to Together's production inference.
The paper is available on alphaXiv.
More from Infra
- Bloomberg columnist urges data center builders to read his overlooked deep-dive — asusarla · 2026-11-05
- Google launches AlloyDB Express: 2-second Postgres provisioning with scale-to-zero for coding agents — rseroter · 2026-11-05
- SGLang Summit 2026 agenda: LinkedIn serves ads model at 10K QPS, NVIDIA brings SGLang to Vera Rubin — DavidBennett__ · 2026-11-05
- A top-tier 28-minute explainer on the structural differences between GPUs, TPUs and Groq's LPU — yacinelearning · 2026-11-05
- Proving a Model's Answer Costs 70-95x the GPU Time of Running It on vLLM — petrusenko_max · 2026-11-05
- AWS shows how to plug calibrated decision models into Bedrock AgentCore Evaluations — AWS ML Blog · 2026-11-05