Brain teaser: strip all attention, MLP and embedding matrices from a Llama 3 — how many trainable params remain?
Sauers_ · x · 2026-10-16
A puzzle from @Sauers: a Llama 3 model has L layers, residual stream width d, and n neurons per layer. Remove every attention matrix, every MLP matrix, and the input/output embedding weights — how many trainable parameters are left? A nice exercise in understanding where Transformer parameters actually live (hint: mostly the norms).
More from Models
- Unverified claim: Apple and Meta's 816B-parameter Muse Spark Open runs on two Mac Studios — TansuYegen · 2026-10-16
- Gemini 4 model card: model eliminated months of trial-and-error on bioweapons research — Puzzleheaded-King584 · 2026-10-16
- Every AI picks 7: Anthropic engineer's rule for escaping model taste convergence — rubenhassid · 2026-10-16
- Lean v4.35.0 ships verified kernels — one written and proven consistent by Claude — geoffreyirving · 2026-10-16
- One flagged seat froze an entire Claude team org — founders share backup plans — RexDouglass · 2026-10-16
- Kimi K3 checkpoint impresses: strong held-out results point to solid pretraining — SonglinYang4 · 2026-10-16