KAIST's KV4-RL: INT4 KV cache quantization speeds RLVR rollouts 1.8x without accuracy loss

kaist-ai · hf · 2026-10-15

Background: asynchronous rollout improves RLVR efficiency, but sustained concurrency concentrates context processing into fewer decode steps, inflating per-step KV workload.
Finding: in matched-workload comparisons, KV cache quantization accelerates far more than weight-only or weight-activation quantization; yet raw INT4 KV rollout can degrade learning even with token-level truncated importance sampling.
Method: KV4-RL combines calibration-free INT4 KV quantization with selective INT8 protection of attention sinks and recent context.
Results: across 400 GRPO iterations of Qwen3-8B-Base, KV4-RL achieves 1.80x cumulative rollout speedup and 1.27x full-iteration speedup over BF16 (1.97x over the final 50 iterations), with final six-benchmark accuracy within 0.5pp of BF16-rollout training for both Qwen3-4B-Base and Qwen3-8B-Base.

More from Infra