Proteus, accepted at NeurIPS 2026, unlocks memory blocks to boost long-context by +8.4 NIAH
behrouz_ali · x · 2026-09-25
Proteus, accepted at NeurIPS 2026, tackles a flaw in linear-time attention: fixed-size memory states expose full capacity from token 1, so early tokens over-consume state and later tokens inherit a full memory.
- Method: partition memory into E blocks and unlock them one by one as context grows; locked blocks are neither read nor written. The state never grows — only its active fraction changes.
- Cost: zero extra parameters and compute, a drop-in gate.
- Results: +0.37–1.05 avg accuracy on language modeling and commonsense reasoning, and up to +8.4 NIAH points at 2× the training context length, with the largest gains at the longest contexts.
The idea: an early bottleneck forces summarization over memorization, leaving fresh capacity for new information to land.
More from Models
- Is the Opus 5.5 hype legit? A dev argues one-shot demos don't reflect real workflows — MrET97 · 2026-09-25
- Why AI writing won't beat humans: writing is anti-inductive and shared models wear out — herbiebradley · 2026-09-25
- Fastino releases 340M open-weight decision model GLiNER2.5-Decide, tops 9 of 17 benchmarks — vanstriendaniel · 2026-09-25
- Alleged Gemini 4 Pro leak table: 7 models vs 17 benchmarks — neovarchTech · 2026-09-25
- MechReason: a 12k-QA benchmark exposing multimodal models' mechanical engineering reasoning gap — AndrewDai · 2026-09-25
- Burkov's TP Weekly #179: GPU rent vs buy, llm-d serving 753B model at 5-10x lower cost — burkov · 2026-09-25