METASTONE launches TokenFleet scheduler, lifting goodput token throughput ~29% on GLM5.3
量子位 · wechat · 2026-10-20
At CCF HPC China 2026, full-stack compute operator METASTONE unveiled TokenFleet, a distributed scheduling framework targeting the coexistence of request queuing and idle capacity in GPU clusters, optimizing request routing, cross-datacenter capacity coordination, and mixed-workload governance around goodput (SLO-compliant token output).
- TokenPilot picks instances within a model pool by jointly estimating KV-cache reuse gains and queuing costs; on tuned GLM5.3 it cut P95 TTFT by 32.4% and P50 TTFT by 71.3%.
- FleetMind handles cross-datacenter routing by aggregating model availability, capacity, and network state, subject to permissions and data-residency constraints.
- SLOMax manages admission, multi-pool queuing, and capacity allocation by workload type; in one GLM5.3 example, total throughput rose only 1.5% but goodput rose 29.4%, with SLO attainment going from 67% to 86%.
METASTONE manages 20,000P+ of compute across 10+ supercomputing centers and previously released the Meta-Infer inference engine.
More from Infra
- OpenJet claims Qwen3.8-27B coding runs 1.9x faster than llama.cpp on one RTX 3090 — JeremyCMorgan · 2026-10-23
- Jensen Huang lays out NVIDIA's full Vera Rubin agentic system: NVL72, MGX, CMX and RTX — rohanpaul_ai · 2026-10-23
- ONNX to Transformers.js: running models directly in your browser — JosephJacks_ · 2026-10-23
- MiniMax M3.1 Flash free for 3 days on 0G, with 1M context and 512K output — MiniMax_AI · 2026-10-23
- "We bet on Cloudflare and it paid off": R2 and D1 at billion scale — dinasaur_404 · 2026-10-23
- Public data is drying up; private training data is turning into a browsable catalog — SucceededMind · 2026-10-23