Google Research proposes RCDL: replacing weight matrices with vector databases
kaixhin · x · 2026-11-04
At Sakana AI's FIG seminar, Johannes von Oswald (Google Research PI group) presented "Retrieval-Centric Deep Learning in Growing Nonparametric Neural Networks" (arXiv:2610.03858). Building on the duality view that GD-trained linear layers implement linear attention over training data, the paper stores a key-value pair per training point instead of compressing data into fixed weight matrices, retrieving via attention at inference to form a growing network. Since naive kernel swaps don't optimize principled, they derive functional gradient-based learning rules for RBF and softmax-like kernels, establishing the RCDL paradigm with promising image-classification results, plus connections to mech interp and optimizer design.
More from Research
- Weight-editing author: gains aren't real knowledge — Gemma math didn't improve, but Qwen3 issue is real — lateinteraction · 2026-11-04
- rasbt questions Qwen3 weight-editing gains: 50 GRPO steps already lift MATH-500 from ~15% to ~50% — rasbt · 2026-11-04
- NVIDIA releases Dory1.0-4B, a looped recurrent LM beating larger models — nvidia · 2026-11-04
- Running controlled LLM experiments with Codex CLI: disable every tool via config — lateinteraction · 2026-11-04
- Claude Opus 5 surgically edits ~1% of ProteinMPNN weights into SolubleMPNN-like behavior — AllThingsApx · 2026-11-04
- ESMFold author: AlphaFold-style covariation could unlock small-molecule prediction — sokrypton · 2026-11-04