Solar Open 2’s pretraining trims 20T tokens to 10T and stretches context to 1M
keunwoochoi · x · 2026-07-26
- This post focuses on Solar Open 2’s pre-training pipeline: selective weight transfer from Solar Open 1, then about 12T tokens across three stages.
- The data pipeline refines an initially cleaned 20T-token pool down to 10T, using quality scoring, duplicate removal, and a mixture ratio tuned to favor real data, code, math, and English.
- Stage 4 extends context to 1M tokens and merges checkpoints after observing performance dips during length expansion.
Related event: Upstage Releases Solar Open 2: A 250B Sovereign LLM(6 posts)→
More from Models
- Fork builds engine to run Qwen at ~400 tok/s on three used V100s — jfinkpottery · 2026-10-27
- Meta joins frontier labs reporting CoT controllability evals in Muse Spark 1.3 card — tomekkorbak · 2026-10-27
- The tokenizer tax: AI models cost more when you don't use English — postalex · 2026-10-27
- Meta's Muse Spark 1.3 System Card Relies on Four Tübingen Benchmarks — maksym_andr · 2026-10-27
- ChatGPT Pro 500 Spotted: £445/Month Tier With Shared Weekly Usage Limits — yoimnotkesku · 2026-10-27
- Experts Clash: Today's Models Dwarf GPT-4, But Journal Delays Lock Out New LLMs — DeryaTR_ · 2026-10-27