ModernBERT's 8192 context: 18 of 28 layers are actually 128-token sliding windows
AIQuanting · x · 2026-09-21
In a technical exchange over ModernBERT-style long-context claims, AIQuanting notes that while the README recommends raising maxlen to 2048/4096/8192, changing the config doesn't alter the attention structure: 18 of the model's 28 layers use 128-token sliding window attention, with only 10 full-attention layers. So most of the stack reads locally even with 8192 tokens available, complicating the 'supports 8192' framing.
Related event: Debate Erupts Over ModernBERT-style Model's Long-Context Claims(5 posts)→
More from Models
- YC-backed Monte launches post-training platform to build proprietary models from your agents' real work — ycombinator · 2026-11-12
- Leak Claims OpenAI Will Launch Entire GPT-6.2 Family Tomorrow to Counter Opus 5.5 and Fable 5.2 — heypearlai · 2026-11-12
- ARC-AGI-3 leaderboard: models score 90%+ but at $10K+ per task — haider1 · 2026-11-12
- LMArena Launches Agent Arena for Head-to-Head Agent Evaluation — arena · 2026-11-12
- LLMs lack hyperlocal municipal data, explains disputed Texas power outage debate — csuwildcat · 2026-11-12
- Claude API's per-message effort lets you switch reasoning levels without breaking prompt cache — lyraxana · 2026-11-12