Hybrid attention matches full attention on long context, 'just cheaper,' says author
antoine_chaffin · x · 2026-09-21
Responding to AIQuanting's observation that 18 of the model's 28 layers use 128-token sliding windows with only 10 full-attention layers, author antoinechaffin says it's not an issue: long-context performance is equivalent to full attention across every layer — 'it's just cheaper.'
Related event: Debate Erupts Over ModernBERT-style Model's Long-Context Claims(5 posts)→
More from Models
- Why Is Inkling 2 Small Trained on So Few Tokens Yet Still Competitive? — georgejrjrjr · 2026-11-12
- OpenAI's five biggest failures laid out: pretraining decline, alignment fears, enterprise lost to Claude — teortaxesTex · 2026-11-12
- Fable 5.2 Hits 41.3% on Remote Labor Index, Up From 2.5% a Year Ago — alliekmiller · 2026-11-12
- Beaker automates MIT's weight-space programming: one-neuron edit lifts MATH-500 from 7% to 37% for under $4 — VictorSanh · 2026-11-12
- Researcher says Anthropic dodged questions on Mythos false positive rate; 37,966 findings, 2 exploited — OwariDa · 2026-11-12
- Grok spotted building meeting "Listen to Conversation" feature with live transcript — nima_owji · 2026-11-12