Forking gap follows a power law in context frequency; masking rare n-grams shrinks it
ZimingLiu11 · x · 2026-10-27
Concluding his thread, Ziming Liu quantifies forking: the rarer the context, the larger the gap, roughly a power law in frequency (the paper explains why). Masking low-frequency n-gram reads shrinks the gap—SCONE already works this way, with BPE-style mining allocating memory rows only to frequent n-grams. Charts show probe gap following a power law in context frequency F, and online gap under masking thresholds F≤1/2/4/8/200.
More from Research
- Jacobian Lens explained: a 'lie detector' for reading the mind of an LLM — huopak · 2026-10-27
- Preprint reframes enzyme function prediction as conditional generation with Fluxion — LeoTZ03 · 2026-10-27
- No bins needed: new method estimates model calibration error as a drop in risk — BlackHC · 2026-10-27
- AnyRank ranks robot policies reliably with far fewer real-robot trials — pappasg69 · 2026-10-27
- Cohere Labs releases Qwen investigation traces dataset with repro script — cohere · 2026-10-27
- OpenAI claims models helped falsify the Exponential Time Hypothesis — Environmental_Mix22 · 2026-10-27