Interactive diagram makes Self-Attention vs Cross-Attention instantly clear
ProfTomYeh · x · 2026-09-20
Prof. Tom Yeh's "AI by Hand" series adds an interactive diagram contrasting self- and cross-attention. Self-attention: N tokens query each other, giving an N×N score matrix with Q, K, V all drawn from X. Cross-attention: the same N queries attend to a second sequence's M tokens, giving N×M — the only change is taking K and V from a second source E. In both cases Q and K must share the key dimension for the dot product, while V keeps its own dimension. Part 5 of the Attention series, following QKV projection and multi-head attention.
More from Research
- Deep Learning Theory textbook by Roberts, Yaida, Hanin lands on ChapterPal — burkov · 2026-09-20
- AI researcher warns peer review may need credential gates to stop one-shot AI papers — sethlazar · 2026-09-20
- Frank Nielsen's duo Bregman divergence paper unifies KL across exponential families — FrnkNlsn · 2026-09-20
- Gary Marcus doubts AI "pain axis" study; critics note it's just steering vectors, reproducible by anyone — burny_tech · 2026-09-20
- VBVR-Pro: 300 visual reasoning tasks plus verifiable rewards for RL training — 机器之心 · 2026-09-20
- 45 expert scientists stress-test AI reviewers against Nature-family peer reviews — windx0303 · 2026-09-20