Motif 3: Technical Report
Junghwan Lim, Joon Son Chung, Sungmin Lee, Wai Ting Cheung, Gihun Cho, Minsu Ha, Sangho Kang, Beomgyu Kim, Dongseok Kim, Jangwoong Kim, Taehyun Kim, Taewhan Kim, Jeesoo Lee, Jeongdoo Lee, Junhyeok Lee, Dongpin Oh, Hyeyeon Cho, Dahye Choi, Jaeheui Her, Hanbin Jung, Changjin Kang, Minjae Kim, Youngrok Kim, Hyukjin Kweon, Hongjoo Lee, Yeongjae Park, Bokki Ryu
cs.AI
2026-08-10
An open 314B MoE activating only 13.2B per token. Its GDLA attention merges differential noise suppression with MLA's compressed KV; it holds its own on agentic and terminal tasks against much larger models but trails on pure knowledge and scientific coding.
Open large models have spent the last few years closing the gap with closed ones, and Mixture-of-Experts (MoE) lets capacity grow without proportionally growing per-token compute. The problem this report tackles is how, at large scale, to use a more expressive yet more efficient architecture to turn heaps of data and compute into broad, strong capability. Motif 3, from Motif Technologies (a Korean team), is a 314B-total, 13.2B-activated MoE language model.
The centerpiece is Grouped Differential Latent Attention (GDLA), which fuses three ideas. Differential Attention (from Differential Transformer) subtracts one attention distribution from another to cancel patterns shared by signal and noise and focus attention on relevant context. It is symmetric by default, splitting head capacity evenly between signal and noise; the grouped variant (GDA) makes it asymmetric, giving more heads to the signal path and sharing a small set of noise heads, buying stronger signal modeling for less compute. Multi-head Latent Attention (MLA, from DeepSeek) compresses the KV state into a low-rank latent representation, slashing the inference KV cache. GDLA combines differential noise suppression, GDA's asymmetric signal modeling, and MLA's compact inference state, plus a query-dependent output gate. In a roughly-10B controlled experiment, GDLA reaches a loss of 3.2 with 9.2% fewer tokens than MLA.
Other changes: a hybrid 1-full-plus-3-sliding-window attention schedule; modified manifold-constrained hyper-connections (mHC) replacing residual additions with four parallel streams, with the key fix of annealing the post-mapping multiplier from two to one during training to stop residual amplification from piling up activation outliers at depth; Expert-Specific PolyNorm, giving each expert its own polynomial activation coefficients so different experts learn different nonlinear responses; and a multi-token-prediction head for self-speculative decoding. Each MoE layer selects 8 of 384 routed experts per token, plus one shared expert.
Pretraining uses about 12.5 trillion tokens (web, STEM, code, math, multilingual, with extra emphasis on Korean and on legal and financial data). Post-training runs general SFT, then seven specialist teachers (six trained with GRPO reinforcement learning, one software-engineering teacher with SFT), consolidated into one student by Multi-teacher On-Policy Distillation (MOPD). The model is text-only with a 256K context limit.
Against much larger open models (MiniMax-3 428B-A23B, GLM-5.1 744B-A40B, Kimi-K2.6 1T-A32B, Qwen-3.7 Max, DeepSeek-v4-Pro 1.6T-A49B), Motif 3 shines on agentic and tool tasks. It takes 35.3 on tau3-Banking (the table high; the runner-up is 30.1), 94.7 on tau2-Bank Telecom, 74.9 on Terminal-Bench 2.1 (near the top), and 51.5 on the public ITBench-AA subset (the highest available result). This fits the MOPD recipe of folding agentic, tool-use, and software-engineering specialties into one student.
| Dimension | Motif 3 | Table best |
| tau3-Banking | 35.3 | 35.3 (highest) |
| Terminal-Bench 2.1 | 74.9 | 75.0 (Qwen-3.7) |
| SWE-bench Verified | 76.2 | 80.4 (Qwen-3.7) |
| GPQA Diamond | 83.4 | 92.9 (MiniMax-3) |
| SciCode | 40.6 | 53.5 (Kimi/Qwen) |
The weak spots are equally clear: pure knowledge (GPQA Diamond 83.4, far below MiniMax-3's 92.9), scientific coding (SciCode 40.6), and specialized scientific reasoning (CritPt 6.6) all trail the strongest models. On calibration there is an interesting point: AA-Omniscience accuracy is a modest 30.1, but its non-hallucination score of 71.6 is among the table highs (DeepSeek-v4-Pro sits at 5.9), suggesting a conservative balance between answering correctly and refusing to fabricate.
For a 13.2B-activated model to hold its own against 1T-class models on agentic and terminal tasks shows that aggressive sparsity (8 of 384) plus targeted multi-teacher distillation is a viable path. GDLA's fusion of differential noise suppression with KV compression is useful for deployments that need long context and low memory. For anyone watching non-English (especially Korean) open ecosystems, this is also a real measured sample from a national foundation-model effort.
It is a text-only model, so tasks needing visual input are out of reach, and the authors list this first. GDLA's advantage is validated only in roughly-10B controlled experiments; whether it holds at the full 314B is not directly compared. Training and evaluation do not cover the full diversity of real tasks, domains, languages, and deployment conditions, and the underrepresented tasks (scientific coding, specialized scientific reasoning) do show up in the scores. The 256K context is supported, but long-horizon agents need state tracking, planning, and recovery over far longer trajectories than evaluated here, which the authors concede is still well short.