MIT Interactive Diagrams: From Attention to Mixtral and DeepSeek-V3 Architectures

vtabbott_ · x · 2026-10-03

A user shares an interactive Transformer architecture diagram tool from MIT (zardini.mit.edu) that lets you explore the internals of many models:

Building blocks: scaled dot-product attention with its training step, causal self-attention with weights and a residual connection, multi-head attention, and grouped-query attention (GQA);
Classic models: Attention Is All You Need (2017), Mixtral-8x7B sparse MoE (2023), DeepSeek-V3 latent attention + MoE (2024);
Modern models: Kimi K3's delta attention and attention residuals, GLM-5.3's MoE + sparse attention, DeepSeek-V4.1-Flash, and MiMo-V2.6-Pro's sliding-window attention + MoE;
Interactivity: switch between Decode and Cached modes to see where the KV-cache should go, with an expressible relationship between prefill and inference forms; toggle quantised vs unquantised (FP vs real numbers) views, visualize arrows and broadcasting, and open the accompanying notebook.

The author notes that abstract representations let you derive where the KV-cache belongs.

More from coding & agent