Together triples Step 3 serving throughput with 8 stacked inference optimizations
togethercompute · x · 2026-11-08
Together AI optimized its serving stack against StepFun's TTFT and TPS targets, lifting Step 3 throughput 3.47x. Agents helped search across MoE routing, scheduling, kernels, speculative decoding and sparse attention, with every change benchmarked before moving on.
The shared chart breaks down 8 compounding optimizations: MoE backend + fused router/top-k (+40%), scheduling + batching (+28%), decode kernel fusion (+12%), speculative decoding (+15%), context-parallel sparse attention (+15%), FP8 prefill (+12%), prefill kernels (+8%), and block-union sparse attention (+8%).
More from coding & agent
- AI Lab Reportedly Fired Engineer for Using AI Agents to Auto-Merge Non-Prod Code — Yuchenj_UW · 2026-11-08
- prips + hakrevdns: a handy combo for enumerating IPs and reverse-resolving them to hostnames — kingslyj · 2026-11-08
- MCP tool descriptions silently change — and none of the 4 major AI coding tools warn you — abhiramreddykalluru · 2026-11-08
- A simple prompt coda stops coding agents from constantly asking for your approval — doodlestein · 2026-11-08
- Apple fumbled the OS-level personal agent opportunity, leaving browser-based agents as a workaround — omooretweets · 2026-11-08
- Stanford's Memorilla compresses LLM memory to 16 tokens, 4212x fewer and 27x faster — _AustinCalvert_ · 2026-11-08