Quiz: how many trainable params does GPT-2 use just for embed and unembed?
Sauers_ · x · 2026-10-17
A short technical quiz from Sauers: GPT-2 embeds all V possible tokens into vectors of length E at the start of its network and unembeds the final hidden state into token logits at the end. Counting only these two steps, the trainable parameters total 2×V×E (e.g. 77M for V=50257, E=768), noting that with weight tying the embedding matrix is typically shared between the two steps.
More from Research
- Claude cracks 25-year-old seeds bounty, its internal doubting-the-simulation reasoning goes viral — rickasaurus · 2026-10-17
- Citations are oddly asymmetrical: unlimited times cited, but capped references — arjunrajlab · 2026-10-17
- Unlearning verdicts flip with learning rate: RMU vs KL Min comparison — _vztu · 2026-10-17
- Fine-tuned fake knowledge collapses in one epoch, pre-trained knowledge fades far slower — _vztu · 2026-10-17
- HySparse 2 called 'DeepSeek-coded' as SWA becomes standard in sparse attention designs — eliebakouch · 2026-10-17
- SAEs Need Far Less Training Data Than Assumed — Unless You Need Feature Reproducibility — Aikyam-Lab · 2026-10-17