Understanding Transformer Core by Hand

ProfTomYeh · x · 2026-07-17

What this is about

The author breaks down a Transformer block using a "hand-calculated" approach, aiming to let readers walk through the forward pass themselves to understand its internal structure.

Core Conclusions

Attention weighting is responsible for mixing information across different positions, allowing a token to "see" the features of neighboring positions.
FFN (Feed-Forward Network) is responsible for mixing feature dimensions within the same position.
Together, these two parts constitute a complete Transformer block. The author emphasizes that other components (like positional encoding, multi-head, layer norm, skip connections, etc.) primarily enhance stability and scale, rather than being these two core computations themselves.

Demonstration Steps in the Article

Provide input features for 5 positions.
Generate the attention matrix via query-key.
Apply attention weights to the input to get a representation mixed across positions.
Feed it into the first layer of the FFN to expand the feature dimensions.
Pass through ReLU to zero out negative values.
Pass through the second layer to compress the dimensions back, and the output moves to the next layer.

Author's Takeaway

Attention mixes information horizontally.
FFN processes features vertically.
The expressive power of the Transformer comes from the repeated stacking of these two mechanisms.

Original post →

Understanding Transformer Core by Hand

More from coding & agent