Proteus, accepted at NeurIPS 2026, unlocks memory blocks to boost long-context by +8.4 NIAH

behrouz_ali · x · 2026-09-25

Proteus, accepted at NeurIPS 2026, tackles a flaw in linear-time attention: fixed-size memory states expose full capacity from token 1, so early tokens over-consume state and later tokens inherit a full memory.

Method: partition memory into E blocks and unlock them one by one as context grows; locked blocks are neither read nor written. The state never grows — only its active fraction changes.
Cost: zero extra parameters and compute, a drop-in gate.
Results: +0.37–1.05 avg accuracy on language modeling and commonsense reasoning, and up to +8.4 NIAH points at 2× the training context length, with the largest gains at the longest contexts.

The idea: an early bottleneck forces summarization over memorization, leaving fresh capacity for new information to land.

Original post →

Proteus, accepted at NeurIPS 2026, unlocks memory blocks to boost long-context by +8.4 NIAH

More from Models