Token embeddings matter far more to value vectors than norms suggest
teortaxesTex · x · 2026-10-28
Researcher Muyu He decomposed every Wv/Wq/Wk matrix in a 780M-parameter transformer by singular-value-aligned input directions, then measured how much each prior-layer output (including token embeddings) projects onto those directions.
Key findings: although token embeddings don't stand out by L2 norm, they project unusually strongly onto the top input directions of Wv matrices across all layers — while Q and K show no such special dependence. This corroborates his earlier EMNLP paper (arxiv 2606.02780) based on ablations. In AttnRes-style models, even large per-layer value weights can't upweight embeddings, since embeddings are normalized to constant norm before layer 1 — revealing a fundamental transformer design constraint: token embeddings are forced to have tiny norms, and value matrices can't get them.
More from Research
- Researchers clash: neuron steering 'not practical', contrastive-pair control vectors work better — burny_tech · 2026-10-29
- Artificial Analysis V5 index shifts to harder benchmarks; reconstructed difficulty scale puts GPT-4 near zero — scaling01 · 2026-10-29
- Harmonic's Aristotle Proves 185 Properties of a 12.2M-Gate Chip Block, Cuts Power ~26% — satnam6502 · 2026-10-29
- Baidu Open-Sources Unlimited OCR 2: 270-Page 1899 Drug Manual Parsed in One Request — MaziyarPanahi · 2026-10-29
- HelixCove: A Dataset Capturing How Work Unfolds Across an Organization — niloofar_mire · 2026-10-29
- BioBench Joins Fireworks' Intelligence Index: 1,337 Tasks Testing Bio Agents and Safeguards — kenbwork · 2026-10-29