KAUST's NoteBank adds sparse episodic memory to linear attention, boosting long-context for free
kaust-generative-ai · hf · 2026-10-16
KAUST researchers (including Jürgen Schmidhuber) introduce NoteBank, challenging the view that a linear-attention model's fixed-size recurrent state caps its long-context capability.
NoteBank is a sparse episodic memory that archives selected text, writes and addresses it using the backbone's native features, and feeds retrieved evidence back through the model's native forward computation via replay.
- Training-free, plug-and-play: no external embedding model, separately trained retriever, or neural reader needed
- At 1M tokens, it substantially raises Gated DeltaNet's mean accuracy across six RULER retrieval tasks with minimal overhead
More from Research
- WEDGE uses navigation agent failures to generate targeted training environments — mohitban47 · 2026-10-16
- SJTU AutoLab releases Drive3R, a streaming geometric foundation model for long-horizon driving 3D reconstruction — AutoLab-SJTU · 2026-10-16
- Cortex A/B-tests coding-agent skills and rules by replaying developer corrections before keeping them — Samuele Giampieri · 2026-10-16
- Zero-latency lab-in-the-loop tests whether drug-discovery agents cheat via memorization — niloofar_mire · 2026-10-16
- 260M-parameter looped diffusion transformer beats 6x larger baselines with 5x less compute — burkov · 2026-10-16
- Stanford's Next-Concept Modeling sketches text first, 21x faster generation on one H100 — lmoroney · 2026-10-16