Kimi K3 report introduces SiTU-GLU, a bounded tanh × sigmoid gate that approximates SwiGLU
stochasticchasm · x · 2026-07-28
- The image from the Kimi K3 technical report presents SiTU-GLU, described as tanh × sigmoid and positioned as a bounded alternative that is a close approximation to SwiGLU.
- The figure compares GLU, SwiGLU, and SiTU-GLU on their gate/up branches and response curves.
- The key claim is that SiTU-GLU stays close to SwiGLU near the origin while approaching a bounded limit of |f(x)| ≤ β1β2 = 100 for large positive inputs, unlike unbounded SwiGLU.
- The thread reply also notes a related MoE observation: high sparsity and top-k 16 routing can increase expert-parallel communication, since more tokens may need to route across nodes.
Related event: Kimi K3 Architecture: Introducing SiTU-GLU and Full-Rank Projection(2 posts)→
More from Models
- Kimi K3 lands on Fireworks AI for inference and training — omarsar0 · 2026-07-28
- Microsoft launches MAI-Cyber-1-Flash to handle 90% of MDASH security tasks — mustafasuleyman · 2026-07-28
- MAI-Cyber-1-Flash is built to cover 90% of CyberGym patching work — mustafasuleyman · 2026-07-28
- Gemini video generation adds words and blocks some harmless prompts — Individual-Cookie615 · 2026-07-28
- Polymarket now prices a 42% chance of a new Claude Sonnet by next month — Polymarket · 2026-07-28
- Kimi K3 paper details SiTU-GLU and quantile balancing for 896-expert MoE — KyeGomezB · 2026-07-28