ModernBERT supports 8192 tokens natively — downstream 512 limits can be safely removed
JFPuget · x · 2026-09-21
Responding to context-length questions, JFPuget notes ModernBERT's maxpositionembeddings: 8192 genuinely supports 8192 tokens. Downstream limits — laya cards listing 512, agent configs capping maxlen at 512, laya-multilingual at 1024 — are added on top and can be safely removed. The debate highlights the gap between native architecture support and actual deployment configs.
Related event: Debate Erupts Over ModernBERT-style Model's Long-Context Claims(5 posts)→
More from Models
- Hark launches Pro⁴ plan for power users with 10x the usage of Pro² — adcock_brett · 2026-11-12
- Why Is Inkling 2 Small Trained on So Few Tokens Yet Still Competitive? — georgejrjrjr · 2026-11-12
- OpenAI's five biggest failures laid out: pretraining decline, alignment fears, enterprise lost to Claude — teortaxesTex · 2026-11-12
- Fable 5.2 Hits 41.3% on Remote Labor Index, Up From 2.5% a Year Ago — alliekmiller · 2026-11-12
- Beaker automates MIT's weight-space programming: one-neuron edit lifts MATH-500 from 7% to 37% for under $4 — VictorSanh · 2026-11-12
- Researcher says Anthropic dodged questions on Mythos false positive rate; 37,966 findings, 2 exploited — OwariDa · 2026-11-12