Tsinghua paper reveals "competing learning" in hybrid LLMs; staged freezing boosts retrieval

ChengleiSi · x · 2026-11-01

• Tsinghua THUNLP's new paper identifies a Competing Learning phenomenon in hybrid LLM architectures (recurrence + softmax attention, as used in Kimi K3, GLM-5.3-Flash, Qwen3-Flash-Next, Nemotron-3-Ultra): whichever mixer learns retrieval first suppresses the learning signal for the other. Under typical data, recurrent layers learn retrieval faster, so attention rarely gets to learn it.
• Fix: Better Layer Division (BLD) — freeze recurrence early in pre-training, forcing attention to learn retrieval first, then train both jointly.
• Results on models up to 1B: at 750M, RULER accuracy rises from 26.53% to 32.82%, long-context perplexity drops from 12.68 to 8.08, plus gains in loss and common-sense reasoning. Code released at thunlp/competing-learning.

Original post →

More from Models

• Dev uses ChatGPT visuals for conference slides, says image generation is much better — m_wulfmeier · 2026-11-01
• Bitstream diffusion nearly matches autoregressive LMs in new entropy-gated paper — LucaAmb · 2026-11-01
• Pairing Claude with on-device small models: a local-first hybrid AI workflow on a phone — copenhagen_bram · 2026-11-01
• Gemini 4 Argon ranked: #1 in visual understanding, #2 in SWE-Bench Pro V2, but #11 in financial reasoning — Hrstar1 · 2026-11-01
• ChatGPT and Sora limits merged: image generation now eats your chat rate limits — Alternative-Car8221 · 2026-11-01
• OpenAI to train on previously off-limits Daybreak Blue session data, even with training off — The Decoder · 2026-11-01

Models channel →