Sakana AI Introduces Fast-Weight Product Key Memory Method
SakanaAILabs · x · 2026-07-04
Sakana AI's paper "Fast-weight Product Key Memory" (FwPKM) is slated for an oral presentation at the ICML 2026 "Towards Sustainable AI: Continuous Adaptation" workshop. The method combines Test-Time Training (TTT) with Product Key Memory (PKM) to create sparse fast weights, enabling high-capacity memory construction during inference without updating all weights. It offers large storage capacity, sparse access, and supports test-time writing. Under a 128K context length, it achieves about 75% accuracy on the 5-needle NIAH benchmark, and rapidly adapts to new domains after reading 4 million tokens from a target domain, with ΔNLL dropping by up to 0.4.
More from Research
- Could 10k agents discover learning methods beyond backprop, or just tweak existing ones? — SeunghyunSEO7 · 2026-09-11
- Cognition's SWE-2 uses a KKT duality argument in RL to shift the effort Pareto curve — YouJiacheng · 2026-09-11
- VidMap uses RoMa coarse matching on all frames, fine-scale only for keyframes — ducha_aiki · 2026-09-11
- Bug Hunt Bench author: leaderboard noise is about 2-3 points — PawelHuryn · 2026-09-11
- PNAS paper shows a tiny billiard-ball system is a universal computer — undecidability lives in two dimensions — eigensteve · 2026-09-11
- New paper: Absolute pose estimation from affine cues and gravity direction — ducha_aiki · 2026-09-11