Together Compiler sets new concurrency TPS records across models with just 2 nodes and 24 hours
simran_s_arora · x · 2026-11-05
Together's team reports that the Together Compiler sets state-of-the-art per-user P90 concurrency TPS results across every tested model/hardware setup, building on the best InferenceX engine configuration. The pipeline uses a parallelism cost model to pick effective parallel configs (several differing from existing InferenceX choices), then runs the compiler for 24 hours on at most 2 nodes per model. Results vs InferenceX best: MiniMax-M3.1-Flash B200 633→1111, DSV4.1-Flash B200 671→813, DSV4.1-Flash GB300 943→1205, GLM-5.3 B200 329→417, K3 GB300 423→450, GLM-5.3 MI355X 463→480 TPS per user.
More from Infra
- Self-hosted AI platform: 3 old GPUs, vLLM per card, and a semantic router as one endpoint — JeremyCMorgan · 2026-11-05
- Marvell down 46% since Jensen Huang called it the next $1T company — GaryMarcus · 2026-11-05
- Megakernel optimization speeds DeepSeek V4.1 Flash decode up to 2.27x on AMD MI325X with no accuracy loss — Electronic-Ebb9771 · 2026-11-05
- Kimi K3.1 Pro (2.8T params, 104B active) lands on Ollama Cloud with zero data retention — ollama · 2026-11-05
- Hugging Face shares its model optimization pipeline: profile, torch.compile, ship open kernels — ariG23498 · 2026-11-05
- Reddit proposal: use agents to digest fragmented custom inference engines into reusable building blocks — nasone32 · 2026-11-05