Pedro Domingos reminds: every model learned by gradient descent is approximately a kernel machine
pmddomingos · x · 2026-10-18
Pedro Domingos re-shared his 2020 paper arguing that gradient descent essentially stores training-example memories in weight vectors via superposition. The paper shows that deep networks trained with standard gradient descent are mathematically approximately equivalent to kernel machines—methods that memorize data and predict via a similarity function. This makes network weights interpretable as a superposition of training examples, with the architecture encoding knowledge of the target function into the kernel, potentially guiding better learning algorithms.
More from Research
- SFT + simple search tool lets Qwen3-4B match a ~1T-param model on diagnosis benchmark — Tim_Dettmers · 2026-10-18
- Automating Experiments Makes Premise Choice the Real Bottleneck in Recursive Self-Improvement — Gauri_the_great · 2026-10-18
- Papers with Code Launches Following Feed to Track AI Tasks, Benchmarks and Authors — NielsRogge · 2026-10-18
- Moritz Hardt: AI benchmarks are fundamentally broken but fixable — asusarla · 2026-10-18
- Attention head acts as a switch: mechanistic interp case flips model answer from No to Yes — CatAstro_Piyush · 2026-10-18
- Harvard's open-source ML Systems textbook hits 29.1K stars on GitHub — mdancho84 · 2026-10-18