Shanghai AI Lab's 8.9B "Next Concept Prediction" model hits same loss with 51.3% tokens
jiqizhixin · x · 2026-09-30
Shanghai AI Laboratory and SJTU's LUMIA Lab released NCP-ArchPreview, billed as the first 8.9B discrete latent-space foundation model, topping HuggingFace Daily Papers. It replaces pure Next-Token Prediction with "Next Concept Prediction" — predicting high-level concepts in a discrete latent space, echoing the pixel-to-latent shift in image generation and JEPA-style insights. Trained on Dolmo-3's 5.73T open-source corpus, it matches a full-budget model's final pretraining loss using only 51.3% of the token budget, converges 1.95x faster, and finishes 0.091 below the official baseline loss.
More from Models
- Anthropic 'Drops a Banger Gift' for Claude Users, Says Popular AI Blogger — eyishazyer · 2026-09-30
- GPT 6.1 Sol cuts cached pricing 50% while Anthropic holds back models over safety — oran_ge · 2026-09-30
- OpenRouter data: token usage exploding, some open-weight models see 10x spend since January — AccBalanced · 2026-09-30
- GPT-6 Astra makes generating Minecraft mobs trivially easy — Angaisb_ · 2026-09-30
- Sentdex: OpenAI nerfing the $200 plan is just the start, API prices are the real prices — Sentdex · 2026-09-30
- DepthBench paper finds Pre-LN variants hit a depth wall, comparing 10 residual designs — teortaxesTex · 2026-09-30