Together Compiler hits 2.7x throughput gains on Kimi K3 at same per-user speed
togethercompute · x · 2026-11-10
Together launched Together Compiler (TKCC), billed as the first truly continual compilation system for AI inference, built around "Polymorphic Inference."
The motivation: inference workloads are highly heterogeneous — none of the top-10 open-source models on OpenRouter share the same architecture, Together serves trillions of tokens daily across thousands of customers with distinct traffic shapes, and hardware spans H100/B200/B300/NVL72. Hand-tuned compiler optimizations can't scale to this, and agentic optimizers are bottlenecked by experimentation speed.
Benchmarks vs. the best InferenceX configs on NVIDIA B200/B300 at the same per-user speed: Kimi K3 +2.7x throughput, GLM-5.3 +1.8x, DeepSeek V4.1 Flash +1.8x, MiniMax M3.1 Flash +1.5x.
More from Infra
- Frontier lab compute shifts hard: pre-training falls from 67% to 7%, post-training/RL now 55% — akbirthko · 2026-11-10
- Study of 1,600 Data Centers Finds Each Adds ~1,000-1,300 Local Jobs Within 3-5 Years — RachelVT42 · 2026-11-10
- rakyll teases Agent Substrate as 'agentic compute fabric' ahead of KubeCon keynote — rakyll · 2026-11-10
- ngrok AI Gateway adds Liquid AI and 160+ LLM API models; tiny LFM3-1.7B pages for a thirsty plant — JosephJacks_ · 2026-11-10
- ThumbFree: open-source offline voice typing with on-phone Parakeet, sub-second latency — Affectionate-Top7058 · 2026-11-10
- Bloomberg: Nvidia's Jensen Huang Has ~60 Direct Reports and No Named Successor — rohanpaul_ai · 2026-11-10