EMNLP Oral paper traces multilingual representations in LLMs with cross-layer transcoders
ZhijingJin · x · 2026-10-27
- A paper from Jinesis Lab and collaborators (arXiv:2511.10840, authors include Abir Harrasse, Zhijing Jin, Bernhard Schölkopf) was accepted as an EMNLP 2026 Oral.
- Using cross-layer transcoders (CLTs) and attribution graphs, the authors show multilingual LLMs rely on highly similar shared features across languages in middle layers, while language-specific decoding emerges in late layers.
- Models trained without English show the same shared multilingual space. Output language is picked by a small set of high-frequency late-layer features; intervening on them can suppress one language and substitute another.
- A model-diffing experiment attributes non-English underperformance to dim late-layer features, weak middle-layer clusters, and English-biased tokenizers; finetuning strengthens these features and improves performance.
- The team releases 9 multilingual models, 3 finetuned models, and their CLTs.
More from Research
- New paper: LLM data schedules and learning rate schedules should be co-designed — jasondeanlee · 2026-10-27
- What makes research get adopted: important problems, proper evals, and usability — srchvrs · 2026-10-27
- Bitter lesson part 2: few academic inventions matter, and conference publications aren't the key — srchvrs · 2026-10-27
- First-week PhD student lands ARR Area Chair role, exposing peer-review quirks — anmarasovic · 2026-10-27
- EMNLP keynote: industry does read academic NLP papers, it just rarely credits them — srchvrs · 2026-10-27
- UK AISI: under $80 of fine-tuning fully restores filtered bio capabilities in open-weight models — connoraxiotes · 2026-10-27