Is Kimi3 still a Transformer given architectural changes?
khademinori · x · 2026-08-18
Discussion on whether Kimi3 is still a transformer given its architectural changes. It argues that if the definition encompasses attention + residuals + normalization + FFN, then it remains one, despite swapping standard MHA, post-LN, and ReLU for RoPE, RMSNorm, SwiGLU, GQA, sliding-window, and hybrid layers.
More from Models
- Best damage control for OpenAI is releasing Astra, says Bindu Reddy — bindureddy · 2026-08-18
- Qwen3.8 27B defaults to xhigh reasoning to maximize benchmark performance — frontsideair · 2026-08-18
- Sol 5.6 Ultra shows repetitive outputs, likely RL fried — Miles_Brundage · 2026-08-18
- Gemini Flash generates 3D physics code in minutes — AI_Andrew · 2026-08-18
- Karpathy defines the LLM 'Cognitive Core': On-device, tool-using, and trainable — cephaloform · 2026-08-18
- Grok's Awkward Output Draws Mockery: "My Wife Wouldn't Approve" — moultano · 2026-08-18