Explaining LLM Forking: Mechanism and Mitigation
Liu Ziming explains the LLM 'forking' mechanism, where overcoding lets an n-gram table hijack context signals and supply about three-quarters of the final residual stream norm, and notes it also emerges during post-training replay, mitigable by freezing the table.
2026-10-27 ~ 2026-10-27 · 2 related posts
- Episode 1: KAN author Ziming Liu uncovers "forking" overfitting and warns against AutoResearch(2026-10-26, 7 posts)
- Episode 2: Explaining LLM Forking: Mechanism and Mitigation(2026-10-27, 2 posts)
- How forking happens: n-gram tables split logits into non-interacting subspaces — ZimingLiu11 · 2026-10-27
- Forking returns in post-training replay; freezing the n-gram table helps — ZimingLiu11 · 2026-10-27