Solar Open 2’s pretraining trims 20T tokens to 10T and stretches context to 1M
keunwoochoi · x · 2026-07-26
- This post focuses on Solar Open 2’s pre-training pipeline: selective weight transfer from Solar Open 1, then about 12T tokens across three stages.
- The data pipeline refines an initially cleaned 20T-token pool down to 10T, using quality scoring, duplicate removal, and a mixture ratio tuned to favor real data, code, math, and English.
- Stage 4 extends context to 1M tokens and merges checkpoints after observing performance dips during length expansion.
Related event: Upstage Releases Solar Open 2: A 250B Sovereign LLM(6 posts)→
More from Models
- Unreleased 'GPT 6 Sol' model spotted in OpenAI's API — SteveEricJordan · 2026-09-11
- DeepSeek update keeps cache hits mid-conversation, cuts costs 36.6% — teortaxesTex · 2026-09-11
- AI Sextet offers 6 models free and unlimited for 14 days, including DeepSeek and Qwen — airesearch12 · 2026-09-11
- BullshitBench update: GPT-6-Astra beats all prior OpenAI models but still trails Anthropic — scaling01 · 2026-09-11
- Astra Scores 83% on GauntletBench, First Computer-Use Agent to Beat Human Baseline — ducha_aiki · 2026-09-11
- Kimi K2.8 Preview rolls out: near-K3 coding performance, 1M context for all tiers — teortaxesTex · 2026-09-11