Ziming Liu's "forking" phenomenon: over-encoding designs trigger sudden phase-change overfitting
ZimingLiu11 · x · 2026-10-27
Ziming Liu is publishing a thread on the "forking" phenomenon: training curves that look like forks, with periodic sudden phase changes—a special form of overfitting. It arises with over-encoding designs like DeepSeek V4.1's Engram module: the n-gram table offloads the context signal, supplying 3/4 of the final residual-stream norm and lowering training loss, but splitting continuation logits into nearly non-interacting subspaces. Under data replay each subspace sharpens what it saw and pushes down what it didn't, with nothing pulling back—so overfitting arrives suddenly. All injection routes (input, V, y) induce it.
More from Models
- GPT-6.1 Terra cuts factual error rate from 22.3% to 4.3%, an ~81% drop — haider1 · 2026-10-27
- GPT-6.1 Terra lands: early users praise speed, value and subagent performance — BenBajarin · 2026-10-27
- Leaked GPT-6.1 Terra benchmarks: cheap but 'boring', still far behind Astra and Claude — Angaisb_ · 2026-10-27
- Artificial Analysis Ships Intelligence Index v4.3.2 With GPT-6.1 Terra Comparison — ArtificialAnlys · 2026-10-27
- GPT-6.1 Terra cuts hallucination rate by 50 points while lifting accuracy 2 points — ArtificialAnlys · 2026-10-27
- GPT-6.1 Terra scores 48 on AA Coding Agent Index at one-third the cost of GPT-5.6 Terra — ArtificialAnlys · 2026-10-27