NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction
The Intern-NCP Team, :, Jiaqi Cao, Chiyu Chen, Shuang Cheng, Xu Cheng, Beiya Dai, Yufan Feng, Kewen Ge, Ruijun Ge, Jiayi Huang, Yang Jiao, Dahua Lin, Zhouhan Lin, Yifan Liu, Yuliang Liu, Biqing Qi, Mowen Ruan, Junzhe Shen, Yunchong Song, Hao Sun, Zhongbo Tian, Yixuan Wang, Rubin Wei, Jiaxin Xiong, Kangyu Yang, Qian Yao, Qi Zhang, Bowen Zhou
cs.CL
2026-09-10
An 8.9B latent LM jointly predicts tokens and discrete concepts. On 5.73T Dolma-3 tokens it matches OLMo-3-7B's final loss at 51.3% of the data, +2.45 overall and +5.99 GSM8K.
Autoregressive LMs still predict the next token. Multi-token prediction looks a few steps ahead, but the target remains a surface word. Latent diffusion already showed that moving generation into a more abstract space can buy efficiency. In language, JEPA-style prediction of future representations has rarely been trained at trillion-token scale.
NCP-ArchPreview tests whether discrete concepts learned from the model's own hidden states can be first-class prediction targets, trained jointly with next-token prediction, without breaking standard autoregressive decoding. The control is OLMo-3-7B on the same Dolma-3 data.
The backbone is split into a 16-layer Token Encoder, an 8-layer Concept Module, and a 16-layer Token Decoder (8.94B parameters, hidden size 4096). Every four token states are mean-pooled into a continuous concept, then product-quantized into 32 segments with 128 codewords each. The Concept Module predicts the next concept in this discrete vocabulary. The prediction is causally shifted, repeated four times, and added back into the token path.
Training jointly optimizes NTP and NCP. Hierarchical residuals include intra-module links (IRC) and cross-module links from encoder to concept, encoder to decoder, and concept to decoder. Because the concept sequence is about 4× shorter, each Concept Module block has roughly the parameters of a standard Transformer block at about a quarter of the FLOPs.
Pretraining uses 5.73T Dolma-3 tokens at 8,192 context, matching OLMo-3. Mid-training adds about 100B tokens. At inference the model still emits tokens autoregressively. The concept path is extra supervision and internal guidance, not a new generation API.
In Stage-1, NCP reaches OLMo-3-7B's final loss at 51.3% of the tokens, about 1.95× faster, and finishes 0.091 lower. Downstream macro-average rises 46.59→49.04 (+2.45). GSM8K goes 39.27→45.26 (+5.99), PiQA 72.25→80.85 (+8.60). Math average +3.75, code average +2.64.
After Stage-2 the loss gap remains (1.51× to match the baseline's final loss), but the downstream lead shrinks to +0.59 (56.98→57.57). GSM8K is still +3.34 at 83.02. Code average falls 0.65, and HumanEval drops 49.31→45.62. Stage-2 is only about 10% code, so fitting the mixture can hurt underweighted domains.
| Setting | Baseline | NCP |
| Stage-1 macro-avg | 46.59 | 49.04 (+2.45) |
| Stage-1 GSM8K | 39.27 | 45.26 (+5.99) |
| Tokens to match final loss | 100% | 51.3% |
| vs size-matched 8.9B Transformer | 100% FLOPs | 85% FLOPs to approach its loss |
| IsoFLOP vs OLMo-3 | 1.0× | 1.74× compute efficiency |
Ablations add the Concept Module, then residuals, then NCP, and the loss falls at each step. At 1B scale, IRC+CRC cuts loss by 0.0323 versus no hierarchical residual, at +0.051% analytical FLOPs. After pretraining, updating only the 17M VQ module lifts a four-task code average 30.04→32.69; full finetuning drops MBPP+ by 22.49. Injecting concept states into a DFlash2 drafter raises mean accepted length 5.933→6.180 (+4.17%).
This is the largest public demonstration that latent-space prediction can ride along with NTP at 5.73T tokens. The usable artifacts are Stage-1/Stage-2 weights plus checkpoints every 100k steps, a 17M-parameter VQ adapter that leaves the token backbone frozen, and a near-free concept injection for speculative decoding.
It is still an architectural increment. The generation interface is unchanged. Gains sit in pretraining efficiency and some downstream suites. Stage-2 already shows that a lower loss does not automatically become a broad capability lift.
The authors flag two gaps. This preview does not include long-context training, even though a compressed concept path is a natural fit for long sequences. Conversion from loss to downstream scores is unstable: +2.45 after pretraining, +0.59 after mid-training, and tied to the data recipe.
Read the Stage-2 table carefully. Code and ARC-C regress. VQ adaptation keeps code scores up but still loses 0.42 on the general average. OLMo-3-7B is a 32-block 7B model; NCP is 40-block-equivalent at 8.94B. Size-matched and compute-matched ablations exist, and the full model approaches a 40-layer dense Transformer at 85% of its compute, but the headline table is not a strict same-size, same-layout pair.