ByteDance Seed finds DeepSeek retrieval accuracy swings periodically with token position due to KV cache compression

量子位 · wechat · 2026-10-09

ByteDance's Seed team discovered a "Jekyll-and-Hyde" behavior in DeepSeek-V4: adding or removing a few meaningless tokens before a prompt flips the model's answer, periodically, every 4 tokens.

The phenomenon: in a code-completion test using DeepSeek's own FP8 quantization function, padding length mod 4 = 0/1 skewed the model toward the wrong answer (71.3% avg probability); mod 4 = 2/3 flipped it to the right one (91.5%). In 128K needle-in-a-haystack tests, merely shifting the target's position relative to compression window boundaries produced up to a 40.2-point accuracy gap on DeepSeek-V4-Flash-Base.
Root cause: tied to DeepSeek-V4's chunked KV cache compression, which the team names Phase Sensitivity. Even when key and value fall inside the same window, accuracy varies substantially by position.
Reproduction: training Qwen3-0.6B-based models with various chunked compression schemes, every one showed periodic swings matching its compression stride (strides of 4/6/8 → periods of 4/6/8 tokens); full-attention baselines did not. Removing RoPE or using average compression weights didn't help. Attention heads also exhibit Phase Specialization, with different heads favoring different positions.
Mitigation: post-training and architecture updates shrink the gap (V4.1-Flash-0910 down to 6.1 points, period shortened to 2 tokens). Averaged benchmark scores mask the issue, so models using chunked KV compression should be evaluated per compression phase.

Paper: arxiv.org/pdf/2609.36322

Original post →

ByteDance Seed finds DeepSeek retrieval accuracy swings periodically with token position due to KV cache compression

More from Models