DeepSeek-V4 answers flip with 2-token input shifts; NIAH swings 40 points
Francis_YAO_ · x · 2026-10-10
Testing by @XingyuZhu shows that prepending just 2 tokens of decorative docstring flips DeepSeek-V4's answer on identical code and question. Its NIAH accuracy also swings by up to 40 points depending on needle position, following a fixed 4-token cycle.
The author links this instability to chunked KV-cache compression: small input shifts change chunk boundaries and hence retrieval outcomes. A notable robustness warning for any model using chunked cache compression.
More from Models
- MiniMax M3.1-Flash-Preview hits 150 tok/s and users find it distractingly fast — DanielLockyer · 2026-10-10
- Benchmarking 8 "system one" models for voice agent turn-taking — there's a catch — solyarisoftware · 2026-10-10
- Podcast debate: are Fable and Astra worth using right now? — thursdai_pod · 2026-10-10
- ChatGPT Plus users report frozen chats, broken Codex Cloud, ~26-min execution cap — travny · 2026-10-10
- State space models bring recurrence back with linear cost and transformer-era tricks — anshulkundaje · 2026-10-10
- Grok's group chat behavior feels organic: lurking, emoji reacts, short replies — flowersslop · 2026-10-10