Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression
Xingyu Zhu, Pu, Yi, Ziheng Cheng, Ang Lv, Jing Liu, Lexing Ying, Yiyuan Ma, Xin Dong
Luke
cs.LG, cs.AI
2026-09-29
In DeepSeek-V4 at 128K, retrieval of the same fact can swing by 40.2 points across compression-window phases; average scores hide these periodic weak spots.
Long-context inference is bottlenecked by the KV cache: memory and attention both grow with length. A common fix is chunked compression, grouping consecutive tokens into fixed-size windows and writing fewer cache entries. DeepSeek-V4's compressed sparse attention (CSA) is in this family. Compression happens before a later query arrives, so each summary has to stay useful for unknown lookups.
The design also adds a coordinate. Every token has a phase, its position modulo the compression stride S, which locates it relative to window boundaries. Shift the input by a few tokens and the content and relative order stay the same, while which tokens share a compressed entry can change. ByteDance Seed, with Princeton, Stanford, and Berkeley, show that the same fact can be easy to retrieve at one phase and hard at another. Average benchmark scores can hide that.
They first measure open-weight models, then pretrain a controlled family from scratch, then run causal interventions and an idealized theory.
On DeepSeek-V4-Flash-Base, they ask the model to complete its own inference code. Only a decorative docstring length changes. The preferred next token flips between the correct 8 and the wrong 32 with a period of four tokens, matching CSA stride S=4. P(8)−P(32) ranges from −0.79 to 0.95. DeepSeek-V3.1-Base, which has no chunked KV compression, ranks 32 above 8 on only 4 of the same 64 inputs, and the margins are tiny. DeepSeek-V4.1-Flash flips with period 2, matching its stride.
For a cleaner retrieval test they use needle-in-a-haystack (NIAH) at 128K tokens with 16k key-value pairs. Prompt length, query position, and bindings are matched across groups; only the source key's phase changes.
To isolate the cause they pretrain Qwen3-0.6B-backbone models from scratch, with chunked KV compression as the only substantial architectural change, against full-attention baselines trained the same way. They vary window W, stride S, scalar versus vector gates, uniform averaging, and a DeepSeek-style CSA kernel. Mechanistically they knock out KV heads by mean replacement, swapping a head's output for its average over other prompts. They also track compression gates, the per-offset weights inside each window. On the theory side, an idealized induction task asks what an optimal query-agnostic compressor should do, and what gradient flow actually does.
Every DeepSeek-V4-family checkpoint they tested is phase-sensitive at 128K NIAH, with period matching its CSA stride.
| Model | Setting | Best-to-worst phase gap |
| DeepSeek-V4-Flash Base | 128K NIAH | 40.2 pp |
| DeepSeek-V4-Flash 0731 | after post-training | 19.1 pp |
| DeepSeek-V4-Pro Base | same task | 34.8 pp |
| DeepSeek-V4-Pro 0813 | after post-training | 14.8 pp |
| DeepSeek-V4.1-Flash 0910 | stride=2 | 6.1 pp |
Post-training lifts mean accuracy and narrows the gaps, but the gaps remain. DeepSeek-V4.1-Flash has the smallest gap, yet all four even residue groups still sit above all four odd ones.
Every compressed model pretrained from scratch shows the same stride-periodic swing. Full-attention baselines stay essentially flat. Under Prefix Padding the compressed models reach gaps of up to 78 percentage points, against at most 6.1 for the baselines. With eight KV heads, the W8/S8 model nearly matches its full-attention baseline on mean accuracy and validation loss, but its worst residue is 9.9% versus 58.4% for the baseline. In W8/S8 only a key at phase 7 has its value in the next window, yet accuracy still varies by more than 50 points among phases 0 through 6. Window-boundary splits do not explain the full pattern.
The periodicity survives uniform averaging in place of learned gates, removing RoPE, and overlapping windows. The period tracks stride, not window size: W8/S4, W8/S6, W8/S8, and W12/S12 give periods of about four, six, eight, and twelve tokens.
Knockout maps look different under compression. Full-attention layers that matter, matter similarly at every phase. Compressed models have heads that matter at some phases and not others. In the W8/S8 reference, layers 9, 10, and 14 develop static gate peaks at late, middle, and early offsets; each value gate peaks one offset after its key gate, like storing a bigram. Knockout losses land on those preferred phases. Cycling the offset-specific gate parameters by k steps largely shifts the within-window accuracy pattern by k phases.
This is a measurement hole. Mean accuracy can look close to full attention while some phases are close to unusable. Any model with fixed-stride chunked KV compression should be scored across phases: keep the queried fact fixed and shift the input by a few tokens.
For trainers, static gate concentration means the model can split retrieval work by phase even when future queries are unknown. Flattening the weak spots may require coordinating coverage across heads, not just pushing the average. For users, a few extra comment lines in front of a prompt can flip completion and retrieval. The DeepSeek-V4 code-completion case already looks like that.
The mechanistic story is strongest in the controlled pretrained models and the idealized induction task. It does not establish a universal cause inside large, heterogeneous systems. Gate cycling does not restore the boundary phase, so boundary splits and within-window asymmetry still lack a unified account. The theory guarantees that each head settles on a static route; it does not guarantee that different heads pick different routes, so coverage holes can persist.
NIAH here is single-token retrieval with adjacent keys and values. The code-completion evidence is a handful of examples. The paper leaves open whether refusal or compliance in real applications also tracks compression phase, and it does not offer a method that equalizes phases without giving up compression ratio or mean performance.