Researchers let a 1B model read in chunks and extend its context with memory

alex_verem · x · 2026-07-25

Researchers from MBZUAI, Princeton, and the Weizmann Institute redesigned how a 1B model reads long text.
Instead of attending to everything at once, it processes 1,024-token chunks and carries forward important information through associative memory.
The setup is explicitly inspired by human memory: chunk-level full attention acts like short-term memory, while the cross-chunk module acts like long-term memory.
The model is trained progressively, starting with short/easy text and gradually increasing difficulty.
The approach reportedly cuts the GPU-memory burden on long documents and avoids the sharp collapse seen in the baseline model beyond its limit.

More from Research