Fudan & OpenMOSS map the trade-off: long-context hybrids learn more, extrapolate less reliably
jiqizhixin · x · 2026-11-17
Fudan University, Shanghai Innovation Institute and the OpenMOSS team released an arXiv preprint, Mechanics of Long-Context Hybrid Models, studying how long-context architectures learn more within a training window yet extrapolate less reliably beyond it. Key findings:
- Positional behavior matters alongside attention type: NoPE attention gives broad aggregation, while sliding-window, RoPE, and gated linear mechanisms introduce distinct positional biases that can complement it.
- Sliding-window hybrids extrapolate well after short-context training, but the advantage can reverse after long-context continuation; larger windows and LongCE help counter overlearning of local context, and linear hybrids strike a different fitting/extrapolation balance.
- Sliding-Window Linear Attention restricts the effective relative-position range at inference (thread truncated here).
A mechanistic analysis directly relevant to hybrid attention design and position-encoding choices for long-context models.
More from Models
- 'Humans Also Read Linearly and Causally, So It's Not a Transformer Issue' — yacineMTB · 2026-11-17
- Models favor their own APIs in code: Opus calls Claude API in 97% of implementations — santiviquez · 2026-11-17
- Search behavior gap: Opus searched in 10/60 answers, GPT models searched every time — santiviquez · 2026-11-17
- 79% of 2,880 LLM API recommendations went to just four big-cloud services — santiviquez · 2026-11-17
- Hy4 Flash hands-on: DeepSeek-V4.1-level quality at ~1/15 the cost, 60% faster — IlovedamemesUwU · 2026-11-17
- Memory eval leaderboards are misleading: OmniMemEval mixes conditions and tunes on its own benchmark — Snoo-bedooo · 2026-11-17