Debate Erupts Over ModernBERT-style Model's Long-Context Claims
On September 21, community members debated a ModernBERT-style model's long-context capabilities. AIQuanting noted that although the README claims maxlen can be raised to 2048, 4096 or even 8192, changing the config does not change the attention structure: 18 of the model's 28 layers use 128-token local sliding-window attention, with only 10 layers performing full-sequence attention. He also questioned the measurement basis: maxpositionembeddings is set to 8192 in the config, yet the model card (context row) lists 512 and the agent config defaults maxlen to 512.
Confirmed
- 18 of 28 layers use 128-token sliding-window attention; 10 layers use full attention (AIQuanting, based on config files).
- maxpositionembeddings is 8192, while the model card lists 512 and the agent default maxlen is 512.
- Model author antoinechaffin responded that hybrid sliding-window + global attention matches full attention on long-context performance, only at lower computational cost.
- JFPuget explained that 8192 is the model's supported upper bound; smaller limits in downstream cards or agent configs are downstream choices, not model limitations.
Why it matters
- Context length is a key selection metric, but inconsistencies between config fields, model cards, and downstream defaults can mislead users, who must distinguish architectural maximum from effective length.
- Whether sliding-window + global hybrid attention truly matches full attention on long-context tasks is the core dispute in low-cost long-context designs; the author's "equal performance, just cheaper" defense has not resolved community doubts about measurement methodology.
2026-09-21 ~ 2026-09-21 · 5 related posts
- Episode 1: Debate Erupts Over ModernBERT-style Model's Long-Context Claims(2026-09-21, 5 posts)
- Episode 2: Model Author Defends Long-Context Claims, Says RoPE Tuning Extends Length(2026-09-21, 2 posts)
Primary sources
- Hybrid attention matches full attention on long context, 'just cheaper,' says author — antoine_chaffin ·
- ModernBERT supports 8192 tokens natively — downstream 512 limits can be safely removed — JFPuget ·
- ModernBERT's 8192 context: 18 of 28 layers are actually 128-token sliding windows — AIQuanting ·
- ModernBERT's 28 layers aren't uniform: 18 slide a 128-token window, only 10 attend fully — AIQuanting · 2026-09-21
- [source] Hybrid attention matches full attention on long context, 'just cheaper,' says author — antoine_chaffin · 2026-09-21
- Config says 8192, model card says 512: which run produced the long-context number? — AIQuanting · 2026-09-21
- [source] ModernBERT supports 8192 tokens natively — downstream 512 limits can be safely removed — JFPuget · 2026-09-21
- [source] ModernBERT's 8192 context: 18 of 28 layers are actually 128-token sliding windows — AIQuanting · 2026-09-21