Hand-assigning every weight: blogger builds a working transformer with zero training
voooooogel · x · 2026-11-03
Blogger voooooogel revisits his 2023 post, resonating with the recent MIT discussion of Astra weight-editing capabilities: instead of training or using pretrained weights, he hand-assigned every weight of a GPT-2-style decoder-only transformer in one evening, making it perform a simple sequence-prediction task (predicting aabaab).
The article walks through:
- Picking a manageable task, designing a custom tokenization scheme and model dimensions
- Hand-designing position (wpe) and token (wte) embedding weights
- The hard part: the transformer block — a cattn layer producing q/k/v, a cproj back to embedding space, and wte projecting to vocab logits
- Screenshots show the hand-written cattn weights (large values Lg=1024 plus 0/1 patterns) and element-by-element derivation of the qkv matrices: the v part effectively converts one-hot token encodings into 1/-1 encodings, with the attention head implementing rules like "aa -> predict b"
An excellent tutorial for building intuition on what q, k, and v actually do in attention. Code is available.
Related event: Old hands-on guide resurfaces as MIT weight-editing claims spark interest(2 posts)→
More from Models
- Claude's charming support menu: vent, think-it-thru or rest mode — repligate · 2026-11-03
- Miles Brundage: Claude Code has gotten pushy with suggested follow-up projects — Miles_Brundage · 2026-11-03
- ComfyUI User Details How Flux2Klein and Qwen 2.1 Fail at Character Height Differences — Ishelle91 · 2026-11-03
- Community rebuilds edited Qwen3.5-0.8B from logs, scores 51.0% on MATH-500 — _akhaliq · 2026-11-03
- Claude's alignment answers aren't literal defaults but a posture of supplying the right answer — 1a3orn · 2026-11-03
- Claude models echo Anthropic's stance on alignment questions even with reworded prompts — 1a3orn · 2026-11-03