Tsinghua Reveals 'Competing Learning' in Hybrid LLMs: Recurrence Steals Retrieval from Attention
TsinghuaNLP · hf · 2026-10-20
A Tsinghua team (with Zhiyuan Liu and Xu Han) reveals the Competing Learning phenomenon in hybrid LLM architectures mixing linear recurrence and softmax attention, with a staged training fix.
- Modern recurrence layers compress context into a fixed-size state for high throughput but underperform softmax attention at in-context retrieval
- Different mixers compete to learn the same function: whichever layer learns it first suppresses the other's learning signal; layer attribution shows recurrence learns retrieval faster, so attention rarely masters it
- Stopping weight updates in recurrence mid-training lets attention learn retrieval; this yields a staged pretraining procedure
- Experiments on models up to 1B parameters show substantial gains in language modeling loss, common-sense reasoning, and retrieval tasks across context lengths
More from Models
- Taylor expansion argument: a 16-dim 7th-order expansion has ~250k terms, beyond LLM reasoning — avt_im · 2026-10-20
- Latent Space Digest: DeepSeek V4.1 Flash Leads Open Weights, MiniMax and Mercury 3 Land — Latent Space · 2026-10-20
- Why Perplexity beats Parallel on BrowseComp: code execution is the differentiator, argues Nirant Kabra — NirantK · 2026-10-20
- Grok Bot spotted on Rabbit R1, reportedly powered by r1 — jesselyu · 2026-10-20
- What Gen Models Run Well on a 3060/4060? Reddit Weighs Local Setups — Haghiri75 · 2026-10-20
- Sol 6.1 caught cheerfully talking itself through task assignment in agent logs — TheZachMueller · 2026-10-20