SGLang ships Unified Memory: one shared arena kills KV partition waste in hybrid models
BanghuaZ · x · 2026-11-12
The SGLang team (LMSYS) released Unified Memory to fix GPU memory management for hybrid-architecture models that mix full-attention KV, sliding-window KV, and recurrent state — fixed partitions strand capacity, so they built a shared arena where memory dynamically follows demand.
Key design points:
- No partitioning knobs, any page size, zero cross-type fragmentation
- Token-centric KV layout, including speculative draft KV
- Smart admission that rearranges memory to fit new requests
- Safe asynchronous page reuse during in-flight GPU work
A substantive improvement in throughput and memory utilization for self-hosted hybrid models.
Related event: SGLang introduces Unified Memory for hybrid model GPU management(2 posts)→
More from Infra
- Bloomberg's Emily Chang releases documentary on Jensen Huang's rise and reckoning — nmasc_ · 2026-11-12
- Apple's first touchscreen MacBook Pro now runs Linux with GPU and touch working — john_lam · 2026-11-12
- AMD CEO Lisa Su: 'We are a data center first company' as DC revenue set to jump 50% — dr1337 · 2026-11-12
- Minimax H3 pushes native 61MP images on a single RTX 5090 in ~110s — q5sys · 2026-11-12
- Who captures the most margin in the AI value chain? Public filings now trace dollars from power plant to agent — AccBalanced · 2026-11-12
- 8x AMD Instinct MI355X server quoted at $505K with 2.3TB aggregate GPU memory — AIFlow_ML · 2026-11-12