Shared KV memory makes Looped Transformers better and 4.7x cheaper: Cornell/IBM paper
yoavartzi · x · 2026-10-28
A Cornell/IBM Research paper shows that in Looped Transformers, only the first recursion needs to write its own KV cache—later recursions can share it. Removing 98.4% of the KV cache in recursions 2–5 of a 1B model actually improves FineWeb-Edu perplexity from 11.39 to 10.32 and lifts 5-task accuracy from 63.4% to 68.2%, while cutting KV memory 4.7x.
Their Looped Prediction Transformer (LPT) sets a new quality-memory Pareto frontier for looped models at 1.5B params, lowering perplexity by 1.12–1.82 vs a matched standard Transformer with 76–79% less context memory; the hybrid H-LPT compares against a gated DeltaNet hybrid LT2. Analysis suggests shared and local memories develop distinct representations, later recursions attend mostly to the shared memory, which also acts as a gradient highway to the first recursion.
More from Infra
- SpaceX eyed as largest VR200 buyer in 2027: 20K racks, ~$300B total — bookwormengr · 2026-10-28
- MySQL recursive CTEs time out at 500M rows; C# BFS answers in 2.7ms vs NebulaGraph 23.6ms — JeremyCMorgan · 2026-10-28
- Token demand on OpenRouter grew 27x to 165T weekly, open models now serve 72% — FinanceYF5 · 2026-10-28
- AI's unusual bet: buying compute for years, selling intelligence by the token — vaibhavbetter · 2026-10-28
- OpenAI's Will Depue says compute locks in US primacy as 'end of human history' nears — willdepue · 2026-10-28
- Malaysia has overtaken Singapore in data center capacity, planned buildout 10x — bytebot · 2026-10-28