Stanford's Memorilla compresses LLM memory to 16 tokens, 4212x fewer and 27x faster

_AustinCalvert_ · x · 2026-11-08

Researchers from Stanford, Toronto, Penn, MIT and others (incl. Jure Leskovec) introduce Memorilla, treating memory as a modality for frozen LLMs.
A learned memory module reads an entire document collection and distills each query's needs into just 16 configurable memory tokens, with model weights frozen.
Results: up to 4212x document-token compression vs full context (NarrativeQA) and 27x lower end-to-end latency (512 docs: 0.74s vs 19.9s on one H100), while improving accuracy on personalization, long-document and knowledge QA — pushing a new Pareto frontier beyond RAG and agentic memory.
Motivation: frozen models forget once text leaves the context window; retraining is impractical and retrieval misses weakly-related facts.
Experiments supervised on Qwen3-8B; code and paper are open.

More from coding & agent