Token embeddings 'looked for' by value matrices but contribute little, analysis finds
RichmanRonald · x · 2026-10-26
Muyu He argues standard transformers harbor a fundamental conflict: value matrices consistently seek token embeddings in their privileged input directions, yet those embeddings end up contributing little to value vectors.
- Method: decompose every Wv/Wq/Wk in a 780M model by singular values to find each matrix's 'looked-for' input directions, then measure how much each previous layer output projects onto them relative to its norm
- Finding: no layer stands out directionally for Q or K, but token embeddings show unusually large projections onto almost all Wv matrices across layers — despite unremarkable norms — while their actual squared-norm contribution to value vectors stays minimal
- Matches his earlier EMNLP paper (arXiv:2606.02780) that reached the same conclusion via ablations and validation loss
More from Research
- Sakana AI hosts Tokyo RSI symposium: AI Scientist in Nature, Schmidhuber keynotes — RobertTLange · 2026-10-26
- HuggingFace's Carbon-X Spots Pathogenic Splice Variant Where Gemini 3.8 Flash Misreads the Gene — LoubnaBenAllal1 · 2026-10-26
- ETH Zurich paper: dynamic evaluation allocation matches 10x-budget model rankings — zouharvi · 2026-10-26
- DuoLM: Fast LLM-Based Personalized Recommendation via Compact Encoder-Ranker — GoogleDeepMind · 2026-10-26
- DISC Learns to Combine Time-Series Dataset Similarity Metrics for Transfer Prediction — Yen-Ku Liu · 2026-10-26
- Adobe's SliderEdit++ Improves Slider-Based Image Editing by 18.5% with New SlideBench — Arman Zarei · 2026-10-26