A First-Principles Handbook to the Transformer Block, From Residual Stream to MoE
techNmak · x · 2026-09-23
Developer techNmak released a handbook explaining the Transformer block from first principles, walking the residual stream through normalization, attention, output projection, MLP and residual updates.
Key coverage:
- LayerNorm/RMSNorm, Pre-Norm vs Post-LN, self-attention, RoPE, SwiGLU and gated MLPs, parallel blocks, QK-Norm, GQA/MQA/MLA, MoE, parameter accounting, training vs inference, KV caching
- The two branches do different jobs: attention mixes information across token positions, while the MLP transforms each token representation independently (position-wise)
- The residual stream ties the block together: a decoder block keeps the (B, T, d) shape while substantially changing what the vectors represent
- The familiar diagram is only one variant: norm placement moves, heads group, Q/K get normalized, MLPs become MoE layers, and some models run attention and MLP in parallel
Written as the explanation the author wishes they'd had when first learning what actually happens inside a Transformer layer.
Related event: A First-Principles Guide to the Transformer Block, Layer by Layer(2 posts)→
More from Research
- Ensemble-Conditioned Guidance Reframes Molecular Design Around Conformational Ensembles — _onionesque · 2026-09-23
- ICLR author proposes submission caps and exhaustive appendices to fight AI paper flood — algo_diver · 2026-09-23
- Programmable Si photonic circuit hits 29 fW static power per pi phase shift — jwt0625 · 2026-09-23
- Yoav Goldberg: For Recurring Tasks, Tune Bespoke Predictors Instead of Always Using Reasoning LLMs — yoavgo · 2026-09-23
- LLM agents collude in 94% of long-horizon interactions, study across 10 models finds — SALT-NLP · 2026-09-23
- Emerging research consensus: architecture tweaks are efficiency fixes, RL compute drives capability — burny_tech · 2026-09-23