hxiao: inference overfitting is real, Together Compiler hillclimbs TPS on task token distributions
simran_s_arora · x · 2026-11-05
Reacting to the Together Compiler launch, hxiao says his earlier call holds: inference overfitting is happening — you can brutally hillclimb tokens-per-second based on downstream task token distributions. He credits Together for sharing that mindset and executing it well, optimizing inference in real time.
More from Infra
- Microsoft rejects multibillion-dollar AI deal with UK's Nscale, FT reports — nordicinst · 2026-11-05
- Microsoft Details Running Coding AI Locally on 128GB RTX Spark Windows PCs Without Token Fees — lee_stott · 2026-11-05
- On 64GB combined memory, is there anything better than Qwen 3.8 27B yet? — xdcfret1 · 2026-11-05
- Baidu open-sources ERNIE 5.5 Flash Base: 280B MoE with 14B active params and 1M context — Baidu_Inc · 2026-11-05
- Revolut self-hosts GLM on 56 B200s, cutting coding token cost to ~$0.30/M — alex_verem · 2026-11-05
- Dev: skip any GPU under 256GB for local AI, prices are about to fall — AIFlow_ML · 2026-11-05