Hunyuan-A13B Technical Report
Tencent Hunyuan Team, Ao Liu, Botong Zhou, Can Xu, Chayse Zhou, ChenChen Zhang, Chengcheng Xu, Chenhao Wang, Decheng Wu, Dengpeng Wu, Dian Jiao, Dong Du, Dong Wang, Feng Zhang, Fengzong Lian, Guanghui Xu, Guanwei Zhang, Hai Wang, Haipeng Luo, Han Hu, Huilin Xu, Jiajia Wu, Jianchen Zhu, Jianfeng Yan, Jiaqi Zhu, Jihong Zhang, Jinbao Xue, Jun Xia, Junqiang Zheng, Kai Liu, Kai Zhang, Kai Zheng, Kejiao Li, Keyao Wang, Lan Jiang, Lixin Liu, Lulu Wu, Mengyuan Huang, Peijie Yu, Peiqi Wang, Qian Wang, Qianbiao Xiang, Qibin Liu, Qingfeng Sun, Richard Guo, Ruobing Xie, Saiyong Yang, Shaohua Chen, Shihui Hu, Shuai Li, Shuaipeng Li, Shuang Chen, Suncong Zheng, Tao Yang, Tian Zhang, Tinghao Yu, Weidong Han, Weijie Liu, Weijin Zhou, Weikang Wang, Wesleye Chen, Xiao Feng, Xiaoqin Ren, Xingwu Sun, Xiong Kuang, Xuemeng Huang, Xun Cao, Yanfeng Chen, Yang Du, Zhen Yang, Yangyu Tao, Yaping Deng, Yi Shen, Yigeng Hong, Yiqi Chen
cs.AI
2026-09-23
Hunyuan-A13B is an 80B-total, 13B-active open MoE with 20T tokens, 256K context, and dual-mode CoT. Slow AIME2024 is 87.3%, BFCL v3 78.3%, LiveCodeBench 63.9%.
Models that handle contest math and coding often activate tens of billions of parameters on every token, which prices out local deployment. Dense 72B models and larger MoEs are expensive for the same reason. Hunyuan-A13B is a sparse MoE with 80B total parameters and 13B active, aimed at matching larger models on reasoning and tool use at a deployable activation budget.
The network is a fine-grained MoE: one always-on shared expert plus 64 same-width routed experts, eight of which fire per token. It has 32 layers, hidden size 4096, FFN width 3072, 32 attention heads with 8 GQA KV heads, a 128K vocabulary, and SwiGLU. Scaling runs showed that zero shared experts hurt, and more than one shared expert flattened quickly, so the design keeps exactly one.
Pretraining reuses the Hunyuan-TurboS cleaning stack, with extra STEM harvesting that yields about 250B high-quality STEM tokens inside a 20T-plus corpus. Training has three stages: 20T at context 4096 (peak LR 3e-4, cosine to 3e-5); a 300B-token fast anneal at context 8192 down to 8e-6; then NTK-aware positional scaling to 32K and 256K.
Post-training is two blocks. Reasoning SFT covers math, code, logic, and science with generative verifiers and a sandbox. Reasoning RL uses GRPO: binary outcome alignment for math/logic/science, a 36-language sandbox for code, 150K prompts, context 24K then 32K, and no KL penalty. A general block then adds writing, multi-turn, instructions, personas, safety, and tools, with a generative reward model plus more than a dozen domain reward services.
Dual-mode chain-of-thought: fast thinking leaves <think> empty; slow thinking writes the steps. Users switch with /nothink or /think; the default is slow. Both modes share one output schema during training.
Among base models, Hunyuan-A13B is compared with Hunyuan-Large (52B active / 389B total), Qwen2.5-72B, and Qwen3-235B-A22B. MMLU is 88.17 versus 88.40 and 87.81. MATH is 72.35, above Qwen3-A22B's 71.84. GPQA is 49.12 versus 47.47. EvalPlus is 78.64 and MBPP 83.86. It beats the team's earlier, much larger Hunyuan-Large on 12 of 14 reported base tests at about one quarter the activation.
Slow-thinking post-training versus OpenAI-o1-1217, DeepSeek-R1-0120, and Qwen3-A22B:
| task | Hunyuan-A13B | Qwen3-A22B | DeepSeek-R1 |
| AIME2024 | 87.3 | 85.7 | 79.8 |
| AIME2025 | 76.8 | 81.5 | 70 |
| GPQA-Diamond | 71.2 | 71.1 | 71.5 |
| LiveCodeBench | 63.9 | 70.7 | 65.9 |
| BFCL v3 | 78.3 | 70.8 | 56.9 |
| ComplexFuncBench | 61.2 | 40.6 | 41.1 |
All four tool-use benches lead the table. Fast-thinking AIME2024 is only 30.6 against 87.3 in slow mode, so the switch is not cosmetic. Long context: PenguinScrolls 87.7, LongBench-v2 55.0 (second to Gemini 2.5 Pro at 60.9), FRAMES 81.1 behind DeepSeek-R1's 85.7. RULER QA averages 76.7 and still 73.9 at 64K-128K, above R1's 65.6. Throughput at A16W16C16, batch 32, 2048-in / 14336-out is 1981.99 tokens/s.
This is a reproducible open MoE report: 13B active, 256K context, a user-visible think switch, and unusually strong tool-calling for the activation size. For teams that want a think-and-call model on a small cluster, 13B active can approach larger MoEs on math and science without 22B active. LiveCodeBench trails Qwen3-A22B by about 6.8 points, so coding is not the selling point. The report landed after the weights were already public; methods are complete, ablations and failure analysis are thin.
There is no standalone limitations section. Slow-thinking LiveCodeBench is 63.9 versus 70.7 for Qwen3-A22B; FRAMES RAG lags R1. Fast versus slow is a control tag, defaulting to slow, so latency-sensitive serving must opt out. STEM harvesting and the 20T pipeline are not independently reproducible. The general RL stack has many reward services and no disclosed weighting to stop them from fighting. Internal benches (LengthCtrl, InsCtrl, ComplexNLU, Word-Task) cannot be checked outside. Throughput is one precision and length setting.