Prof to Hand-Calculate Kimi 3 Architecture: Extreme MoE Sparsity
ProfTomYeh · x · 2026-08-02
Prof. Tom Yeh announced a new 'AI by Hand' seminar focusing on frontier LLM architectures. The session will manually calculate and sketch the math behind Kimi 3 (Moonshot AI), including Kimi Delta Attention, attention residuals, and extreme MoE sparsity (activating just 16 of 896 experts in a 2.8T-parameter model). The lecture will also cover architectural optimizations in Google's Gemma 4 and Alibaba's Qwen 3.6.
More from Research
- New Eval Tests AI Intelligence by Having Models Generate Questions for Other Models — HazanPrinceton · 2026-08-02
- New AI Learns Parkour From Just 30 Seconds Of Video — Two Minute Papers · 2026-08-02
- Physicist Builds Mini-Universe with Cold Atoms to Test the Origin of Time — skdh · 2026-08-02
- Multi-Agent AI Synthesizes 5,100+ Papers to Map Long COVID Mechanisms — alvelda · 2026-08-02
- Proposing Jeffreys-Fisher-Rao Center: Fast Proxy for Symmetrized KL Divergence Centroids — FrnkNlsn · 2026-08-02
- Real AI Usage Reshapes Stock Market: Higher AI Beta Yields Higher Returns — rohanpaul_ai · 2026-08-02