ByteDance: Directional Decomposition Reveals How Transformer Representations Evolve
ByteDance · hf · 2026-09-16
ByteDance researchers treat Transformer representation evolution as functional geometry, decomposing learned updates into parallel and perpendicular components.
- Findings: pretrained models contain substantial parallel components beyond the residual identity path, analyzable in both attention/MLP update space and attention value-aggregation space.
- Targeted edits: exclude-self value-space parallel manipulation is markedly more robust than residual-space or perpendicular counterparts, scaling only the non-self aggregate.
- Compression diagnosis: perpendicular error separates compression methods more clearly than parallel error.
- Training intervention: suppressing full-aggregate parallel components during from-scratch pretraining lowers validation loss and improves downstream averages. Code released.
More from Research
- Sora's Diffusion Transformer explained by hand in 14 steps: prompts enter only as scale and shift — ProfTomYeh · 2026-09-16
- Researcher: any agent acting over long horizons provably has a self-model and world model — chris_j_paxton · 2026-09-16
- Ben Antieau guest post on Terence Tao's blog: mathematics needs both 'fast math' and 'slow math' in the LLM era — littmath · 2026-09-16
- VisTW: a Traditional Chinese VLM benchmark for reading Taiwan — and an eval framework that caught a 36-point bug — piske_usagi · 2026-09-16
- Mind2Dialogue Simulates User Mental States to Train Human-Aware LLMs — Zixuan Wang · 2026-09-16
- NUS Survey Maps Six Roles for Foundation Models Across the Game Lifecycle — NationalUniversityofSingapore · 2026-09-16