Dev explains: model trained at 8k context, RoPE theta tweak extends further
antoine_chaffin · x · 2026-09-21
Model author antoinechaffin answers context-length questions:
- Raising maxlen (README default 512, up to 2048/4096/8192) doesn't hurt long-context performance; 8k is what the model was trained for.
- Architecture detail: the encoder uses 18 sliding-window layers at 128 tokens plus 10 full-attention layers.
- Technically RoPE-based, so you can change theta and train on longer data to extend context.
Related event: Model Author Defends Long-Context Claims, Says RoPE Tuning Extends Length(2 posts)→
More from Models
- a16z's Martin Casado: Google's Carbon is 'really good at code' — xennygrimmato_ · 2026-11-12
- The Zvi opens reaction thread on Gemini 4 Carbon: is it worth extended coverage? — TheZvi · 2026-11-12
- Walden Robotics' Sage model quadruples robot uptime on Toyota factory lines — adnothing · 2026-11-12
- Gemini won't let you keep chat history while opting out of AI training — Google bundles both into one setting — arshdeep · 2026-11-12
- Gemini 4 Carbon Flex tier now costs the same per token as GPT 6.1 Terra — mariofilhoml · 2026-11-12
- Real-world test: Kimi K3.1 Pro costs 5.4x less than Opus 5.5 per task despite 2x list-price gap — Arindam_1729 · 2026-11-12