How forking happens: n-gram tables split logits into non-interacting subspaces
ZimingLiu11 · x · 2026-10-27
Continuing his thread, Ziming Liu explains the mechanism: over-encoding designs like DeepSeek V4.1's Engram module let the n-gram table offload context signal, supplying 3/4 of the final residual-stream norm while training loss drops. But this splits continuation logits into nearly non-interacting subspaces, one per context; under replay each sharpens what it saw and suppresses what it didn't, and nothing pulls it back—overfitting arrives suddenly. All injection routes (input, V, y) induce forking.
Related event: Explaining LLM Forking: Mechanism and Mitigation(2 posts)→
More from Research
- Open Models Match GPT-6 on MedMCQA but Trail 17 Points on Hard Clinical Cases — keunwoochoi · 2026-10-27
- Health-Focused Benchmark Reranks LLMs: Muse Spark Open Jumps from #9 to #1 — keunwoochoi · 2026-10-27
- Open-source Latch compiles PyTorch into training megakernels, 2.75x faster on H100 — sguada · 2026-10-27
- Goodfire commits $10M to a 'Human Genome Project for AI' to reverse-engineer frontier models — CatAstro_Piyush · 2026-10-27
- π0.5 study: raising demos from 10 to 438 lifts Place Ball in Bowl success from 7% to 83% — stingning · 2026-10-27
- HandsOn benchmark: 66 verified tasks and 18,000 human demos for dexterous manipulation — stingning · 2026-10-27