STEPQuant: 6-bit quantization of Delta-rule recurrent states cuts serving memory by up to 68.7%
zju-community · hf · 2026-10-08
STEPQuant is a spatial-temporal post-training quantization framework for Delta-rule recurrent states in linear attention, whose fixed-size states become a major memory bottleneck under concurrent serving; naive low-precision quantization degrades accuracy as errors propagate through successive updates.
Key insight: error impact varies temporally (errors in long-lived memory persist across decoding steps) and spatially (different key rows affect outputs differently, with state magnitudes varying along rows and columns).
Method: STEPQuant allocates precision by error magnitude and memory lifetime, jointly fitting key-row and value-column scales from state distributions and output-error sensitivity.
Results: on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct, 6-bit STEPQuant closely matches FP32-state accuracy and its 4-bit version beats uniform INT8. Integrated into SGLang with optimized kernels, it achieves 5x+ recurrent-state compression and cuts total serving memory by up to 68.7%. Code on GitHub.
More from Infra
- China's electricity glut turns data centers into a solution, as 14nm chips get pressed into service — teortaxesTex · 2026-10-08
- NAVER's DLoop Loops Speculative Decoding Before Verification, Gaining 5-41% Faster Inference Losslessly — naver-ai · 2026-10-08
- Transformer lead times balloon from 500 to 1,120 days, YC partner calls it a startup opportunity — ycombinator · 2026-10-08
- MIT's Christina Delimitrou uses AI to cut data center energy waste and downtime — nordicinst · 2026-10-08
- FT kicks off three-part series on China's breakneck AI infrastructure build-out, from Ulanqab to Shaoguan — zijing_wu · 2026-10-08
- Dev hits 1k tokens/sec prefill at 262K context with hybrid DeepSeek V4.1 Flash build — HankYeomans · 2026-10-08