Sarvam AI serves GLM 5.3 with MXFP8 on vLLM, halving TTFT and doubling throughput
HarveenChadha · x · 2026-11-16
Indian AI company Sarvam AI announced it serves GLM 5.3 with MXFP8 precision on vLLM, halving time-to-first-token and doubling throughput.
Harveen Chadha adds why faster inference matters beyond customers: for evals, faster inference means faster feedback loops; for post-training, faster rollouts in RL translate into more experimentation time and higher research velocity. He says he contributed to token factory work and is applying those learnings to RL.
More from Infra
- DeepSeek Flash on Mac Studio M5 Ultra (512GB): 54 t/s generation with 128K context — vhthc · 2026-11-17
- Peter Diamandis: The Country Building Gigawatts Fastest Will Set the Price of Intelligence — PeterDiamandis · 2026-11-17
- Gemini Managed Agents Adds Self-Hosted Sandboxes: Google Runs the Loop, You Run the Tools — _philschmid · 2026-11-17
- RWKV-8 RPU preview hits 12,500+ tokens/s real-time inference — kastnerkyle · 2026-11-17
- Your AI budget is too generous and too restrictive at once: routing saves, flat caps kill rewrites — ruslansv · 2026-11-16
- Edge inference with llm-d: prefill-decode disaggregation on Qualcomm Cloud AI 100 Ultra — HowDevelop · 2026-11-16