ModernBERT's 28 layers aren't uniform: 18 slide a 128-token window, only 10 attend fully
AIQuanting · x · 2026-09-21
AIQuanting points out that the 28 layers in the model's config aren't uniform: 18 layers use a 128-token sliding window while only 10 attend over the full span, so most of the stack reads locally even with 8192 tokens available.
The follow-up discussion clarifies that maxpositionembeddings: 8192 is just a config field — downstream laya cards list 512, agent configs cap at 512, and laya-multilingual variants list 1024. Extra limits added downstream can be safely removed.
More from Models
- Codex computer use tested: Astra works while Luna and Sol fail on Plus plan — FamilyNP · 2026-09-21
- service_tier=fast rejected on ChatGPT subscription, API-only parameter confirmed — TrickyPlastic · 2026-09-21
- Chinese open-source labs explode on OpenRouter: Moonshot +2425%, Z.ai +1925%, DeepSeek +1000% — FinanceYF5 · 2026-09-21
- Yacine: coding LLMs produce 'total complex garbage' — I still read every line — yacineMTB · 2026-09-21
- Researcher: LLMs write convincing related work, but convincing isn't comprehensive — lucacarlone1 · 2026-09-21
- OpenAI's secret technique for upcoming Astra model sparks security concerns — keviv9 · 2026-09-21