SGLang ships Unified Memory: one GPU memory arena for every token lifetime, no partitioning knob
ying11231 · x · 2026-11-12
LMSYS officially shipped Unified Memory in SGLang (--enable-unified-memory), a months-long memory system rewrite that eliminates one engine configuration knob.
- Hybrid models (full-attention KV, sliding-window KV, recurrent state) previously required a fixed startup-time GPU memory partition; now all state types share one arena and capacity dynamically follows demand
- Any page size with zero cross-type fragmentation via compaction; token-centric KV layout including speculative draft KV; smart request admission; safe async page reuse during in-flight GPU work
- Compatible with speculative decoding, PD disaggregation, and HiCache; team solicits reports of regressions vs. partitioned memory
Related event: SGLang introduces Unified Memory for hybrid model GPU management(2 posts)→
More from Infra
- Gary Marcus amplifies warning to price in end of Jensen Huang's circular-financing demand engine — GaryMarcus · 2026-11-12
- Hippius Hub Positions Itself as Decentralized Hugging Face with One-Line Import Swap — const_reborn · 2026-11-12
- Who could replace Jensen Huang? TechCrunch drafts a shortlist of possible Nvidia successors — TechCrunch AI · 2026-11-12
- SGLang Q4 roadmap: 10+ accelerator families supported, plugin system makes new backends first-class — ying11231 · 2026-11-12
- SGLang ships Unified Memory: one shared arena kills KV partition waste in hybrid models — BanghuaZ · 2026-11-12
- Cerebras explains wafer-scale yield: independent voltage tuning across 84 die regions plus redundant cores — Sethwinterroth · 2026-11-12