Together AI's TKCC agent-compiler beats hand-written Megakernels by up to 1.7x on Llama 1B
realDanFu · x · 2026-11-05
Together AI researchers (Daniel Fu, Simran Arora, et al.) released the paper "Together Compiler: Continual Compilation for AI Inference," describing TKCC, the first truly continual agent loop for AI inference optimization.
- Pain point: models ship fast, architectures diversify, traffic shifts; existing systems face a throughput-fidelity tradeoff — offline kernel-level optimization is low-fidelity with data-dependent primitives like MoE, while end-to-end engine optimization suffers slow launches.
- Key designs: (1) a Tiered Persistent Engine decoupling engine launch overhead from forward-pass computation; (2) a cost model to pick parallelism configurations that seed optimization.
- Extensive ablations on agent context management plus production deployment insights.
- Results: up to 1.7x faster than highly-optimized hand-written Megakernels on the Llama 1B workload, and beats the best open-source engines on latest frontier models (Kimi K3, GLM 5.3, etc.).
- Framed as a first step toward "polymorphic inference," with an accompanying blog on the research agenda.
More from Infra
- Cloudflare now auto-retries Durable Object calls up to 4 times, no retry loops needed — dinasaur_404 · 2026-11-06
- Nasdaq sheds $3.2tn in worst day since April 2025 as Nvidia sinks 9.3% on data center fears — rohanpaul_ai · 2026-11-06
- J.B. Huang's new video explains how LLM serving actually works, from fast to cheap — jbhuang0604 · 2026-11-06
- US BIS extends suspension of affiliates end-user rule through January 10, 2027 — pstAsiatech · 2026-11-06
- Meta's Axia Learns Scaling Laws for Ranking Models from a Few Trial Runs — _reachsumit · 2026-11-06
- US trade deficit ex-data centers is just 0.7% of GDP, lowest since 2000 — soumitrashukla9 · 2026-11-06