The original attention paper discussed the MLP-attention relationship, but only in arXiv v2 and v3
CatAstro_Piyush · x · 2026-10-17
In a thread arguing that every layer is a memory layer whose capacity can be estimated via heuristic interference-suppression arguments, heyyalexwang points out that the original attention paper actually discussed the relationship between MLPs and attention.
The catch: that discussion only appeared in arXiv versions v2 and v3 — the paper is now on v7, so the passage is no longer visible in the latest version.
More from Research
- EMNLP paper: diffusion LMs internally encode latent denoising timestep representations — gsarti_ · 2026-10-23
- KU Leuven and VIB open two Group Leader professorships in AI for Biology, start 2027 — jpillowtime · 2026-10-23
- Subspace Tuning curbs LLM catastrophic forgetting by freezing principal singular directions — scottniekum · 2026-10-23
- Perceptic launches PercepticOS, an AI workspace for 'scientific superintelligence' — nathanbenaich · 2026-10-23
- EPFL's Bosselut to talk long context as test-time learning at ELLIS seminar — andre_t_martins · 2026-10-23
- DRATS: Adaptive Task Sampling Boosts Worst-Task Performance in Multi-Task RL — Cohere · 2026-10-23