ModernBERT's 28 layers aren't uniform: 18 slide a 128-token window, only 10 attend fully
AIQuanting · x · 2026-09-21
AIQuanting points out that the 28 layers in the model's config aren't uniform: 18 layers use a 128-token sliding window while only 10 attend over the full span, so most of the stack reads locally even with 8192 tokens available.
The follow-up discussion clarifies that maxpositionembeddings: 8192 is just a config field — downstream laya cards list 512, agent configs cap at 512, and laya-multilingual variants list 1024. Extra limits added downstream can be safely removed.
Related event: Debate Erupts Over ModernBERT-style Model's Long-Context Claims(5 posts)→
More from Models
- Hark launches Pro⁴ plan for power users with 10x the usage of Pro² — adcock_brett · 2026-11-12
- Why Is Inkling 2 Small Trained on So Few Tokens Yet Still Competitive? — georgejrjrjr · 2026-11-12
- OpenAI's five biggest failures laid out: pretraining decline, alignment fears, enterprise lost to Claude — teortaxesTex · 2026-11-12
- Fable 5.2 Hits 41.3% on Remote Labor Index, Up From 2.5% a Year Ago — alliekmiller · 2026-11-12
- Beaker automates MIT's weight-space programming: one-neuron edit lifts MATH-500 from 7% to 37% for under $4 — VictorSanh · 2026-11-12
- Researcher says Anthropic dodged questions on Mythos false positive rate; 37,966 findings, 2 exploited — OwariDa · 2026-11-12