Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning
Kai Chen, Jifeng Ding, Ning Ding, Jiaye Ge, Lixin Gu, Yicheng Gu, Qipeng Guo, Ermo Hua, Haian Huang, Haozheng Hou, Jie Hou, Xiangyu Hong, Che Jiang, Minxi Jin, Cheng Liang, Dahua Lin, Dawei Liu, Kuikun Liu, Chengqi Lv, Haijun Lv, Han Lv, Ningsheng Ma, Biqing Qi, Jianmin Qian, Shiya Su, Youbang Sun, Huanze Tang, Zhongbo Tian, Hanjing Wang, Rui Wang, Ting Wang, Yi Wang, Baiting Wu, Jun Xu, Bowen Yang, Hui Wang, Weida Wang, Haochen Ye, Jiashuo Yu, Shan Yu, Xiaoyi Yu, Qirui Zeng, Qi Zhang, Ming Zhang, Wenwei Zhang, Bowen Zhou, Xinyu Zhou
cs.AI
2026-08-14
Mobius extracts FFN knowledge into a globally shared Memory and turns Self-Attention into Reasoners that query it iteratively: a 7B model matches a Transformer baseline with 62.6% of the data, and a 35B continual-pretrain beats Qwen3.5-35B while running nearly 4x faster end-to-end.
Transformer has a structural quirk: FFNs store knowledge and Self-Attention does reasoning, but every layer carries its own FFN, so knowledge is scattered and duplicated across dozens of layers. Residual connections flow shallow-to-deep only, so deep attention cannot reach shallow knowledge. The paper's inference: when critical knowledge fails to activate during shallow computation, the model cannot decode a high-information token that round and must grind forward by emitting low-information transition tokens in a long chain of thought. Redundant knowledge storage burns parameters; redundant reasoning chains burn inference compute. The two costs share one root.
Direct backward residual connections would complicate the computation graph and hurt parallelism. Mobius takes an indirect route: every layer shares one oversized knowledge repository, which grants each layer access to all of the model's knowledge.
Mobius-v0 has three parts:
The companion mechanism is dynamic latent reasoning: deliberation, trial-and-error, and refinement move from explicit token generation into optimization of a continuous vector, with compute allocated per token. The authors position this as a synthesis upgrade of Looped Transformers and latent-reasoning research.
Validation runs on two lines: 7B-A1B MoE trained from scratch on 1TB tokens against a matched Transformer, and Intern-S2-Mobius continually pre-trained from Qwen3.5-35B-A3B on 1TB tokens followed by SFT and RL.
| Setting | Result | Comparison |
| 7B from-scratch MMLU | same score with 62.6% of data (1.6x efficiency) | matched Transformer, both 1TB tokens |
| 35B general average | 67.88 | Qwen3.5-35B 65.05 |
| MMLU Pro | 89.05 | 85.31 |
| AIME 2026 | 95.31 | 92.08 |
| Scientific average | 52.14 | 18.20 |
| End-to-end inference | 4x speedup, 1.5x shorter outputs | same problems |
| Linear-algebra CoT | 516 tokens | Qwen3.5-35B 2,364 tokens |
The scientific-task gap (Biology-Instructions 51.40 vs 3.77) is too lopsided to attribute to architecture; it reads as a tilt in the continual-pretraining corpus and deserves a discount.
This is a serious post-Transformer architecture proposal from a team with full training infrastructure, validated the expensive way: a from-scratch controlled pretrain plus a 35B-scale continual pretrain, not a paper-grade toy. If the data-efficiency and inference-speedup numbers survive independent replication, the impact reaches past academia: inference cost is the dominant line item in LLM deployment, and "same quality at four times the speed" hits it directly. Globally shared memory also opens a path to continual learning: updating knowledge without end-to-end retraining is an architectural entry point to the old catastrophic-forgetting problem. All of which rests on the results holding up.
The authors are unusually candid on the two sentences that matter: "whether Mobius merely hacks the problem or genuinely provides a more efficient mode of reasoning remains to be determined," and "the precise mechanism behind Mobius's shorter CoT reasoning remains not fully established." The data-efficiency mechanism likewise stays at hypothesis level, attributed to eliminating cross-layer parameter redundancy, with no ablation or direct evidence.
Gaps from a close read: the 7B comparison reports MMLU alone, which is thin; the scientific-task blowout at 35B is unexplained; the "nearly 4x" speedup aggregates throughput across benchmarks without hardware, batch, and decoding configurations in the main text; and ablations (what happens without the shared Memory, how Reasoner iteration count matters) are absent. Read this as an architecture manifesto with preliminary validation. Until reproduction and third-party evaluation, every number here is a claim.