Researcher Questions Linear Attention: Sparse MLPs May Be a Stronger Baseline
kalomaze · x · 2026-07-30
Independent researcher kalomaze has questioned the current trend of emerging architectures like linear attention and Mamba-esque RNNs. He asked on X: given that language data is discontinuous, if the expressivity gain of linear attention on intermediate tokens is essentially just vector lerping, what real advantage does it offer over a humble sparse MLP?
He pointed out a seemingly trivial, yet widely untested baseline: what if we used basic ResNet MLPs for over 80% of the layers and standard softmax attention for the rest? He argues that the perceived benefits of many new architectures might simply stem from the fact that this fundamental baseline hasn't been thoroughly evaluated. He also noted that architectural effectiveness heavily depends on whether gradient descent behaves non-pathologically given the architectural shape, which is a distinct issue from whether the function class itself is inherently useful.
Related event: Researcher Proposes Minimal MLP Baseline to Replace Most Attention Layers(5 posts)→
More from Research
- When Does Synthetic Data Work? Research Reveals Optimal Ratios and 'Zeta Law' — PTenigma · 2026-07-30
- Applying Jacobian Methods for LLM Contrastive Steering Outperforms Controls — voooooogel · 2026-07-30
- Quadratic Models Surprisingly Accurately Describe LLM Pretraining, Paper Finds — jasondeanlee · 2026-07-30
- Lean as the Ultimate Echo of Principia Mathematica: A Philosophical Divide — doodlestein · 2026-07-30
- Meta & CMU Paper: Agentic Context Management Boosts Long-Horizon Task Performance by 27% — rohanpaul_ai · 2026-07-30
- Scaling Semiconductor Quantum Computers: Qubits Need to Match Classical Transistors — whurley · 2026-07-30