Solar Open 2 uses MoE, linear attention, and no positional encoding
keunwoochoi · x · 2026-07-26
- The post introduces Solar Open 2 as a 250B sovereign LLM and says it is aimed at readers who care about LLM architecture.
- The attached figure shows a 48-layer stack repeating a four-layer pattern: one softmax-attention layer plus three linear-attention layers, each paired with MoE blocks.
- It also highlights 320 routed experts, one shared expert, no positional encoding, and a negative-eigenvalue extension in the linear-attention core to improve state tracking.
Related event: Upstage Releases Solar Open 2: A 250B Sovereign LLM(6 posts)→
More from Models
- Fork builds engine to run Qwen at ~400 tok/s on three used V100s — jfinkpottery · 2026-10-27
- Meta joins frontier labs reporting CoT controllability evals in Muse Spark 1.3 card — tomekkorbak · 2026-10-27
- The tokenizer tax: AI models cost more when you don't use English — postalex · 2026-10-27
- Meta's Muse Spark 1.3 System Card Relies on Four Tübingen Benchmarks — maksym_andr · 2026-10-27
- ChatGPT Pro 500 Spotted: £445/Month Tier With Shared Weekly Usage Limits — yoimnotkesku · 2026-10-27
- Experts Clash: Today's Models Dwarf GPT-4, But Journal Delays Lock Out New LLMs — DeryaTR_ · 2026-10-27