Self-attention as a differentiable hash table: a thread on building QKV intuition

phl43 · x · 2026-10-09

Continuing a discussion of how Raschka's book explains self-attention, prompted by moultano's take that attention is "a differentiable hash table or dict":

The author agrees Raschka leans on a similar analogy but doesn't stress differentiability, since the chapter predates training discussion — though that's obviously the crucial part;
ChatGPT's explanation followed the same lines, again skipping differentiability;
Still, he finds the dict analogy not that useful. More helpful: start from simple self-attention where lookup is purely "semantic" similarity, then show how QKV adds flexibility to attend to non-semantic aspects — which makes especially good sense for multi-head attention.

Original post →

Self-attention as a differentiable hash table: a thread on building QKV intuition

More from Research