Hy4 preview repo goes live: 256 experts per MoE layer, native MTP head, Gated DSA attention
aigclink · x · 2026-08-28
Tencent's Hy4 preview GitHub repo is now public. Architecture details: 770B total / 49B active parameters; a 78-layer backbone where the first layer uses dense FFN and the remaining 77 use MoE — each with 256 routed experts + 1 shared expert, activating top-8 per token. A native MTP layer (10B total, 0.7B active) is built in for speculative decoding. Inspired by DeepSeek and GLM, attention uses Gated DeepSeek Sparse Attention (Gated DSA) with IndexCache for cross-layer sparsity.
The repo includes vLLM/SGLang deployment, finetuning and quantization docs, with bilingual READMEs.
Related event: Tencent Open-Sources Hunyuan Hy4 Preview: 770B MoE with 1M Context(28 posts)→
More from Models
- Nathan Lambert slams Meta's Muse Open License as 'trash' — natolambert · 2026-10-16
- Google launches Gemini 4 Argon to Ultra and Pro subscribers, touting physics and 3D skills — ShikharMurty · 2026-10-16
- Google gives X users early access to test Argon, not just companies — lyraxana · 2026-10-16
- NVIDIA and PrimeIntellect post-train Nemotron 3.5 Super VL, boosting AutomationBench-AA by ~70% — willcb · 2026-10-16
- Grok Voice 3.0 Tops Design Arena Speech Leaderboard, Up 7.4 Points — XFreeze · 2026-10-16
- Robotics LLMs tested on 16 tasks: GPT-6 Astra hits 17.2% success vs Fable's 9.1% — pabbeel · 2026-10-16