Paper accepted to EMNLP analyzes geometry of low-resource language LLM representations
davlanade · x · 2026-08-28
A paper accepted to EMNLP investigates the geometric properties of hidden representations across 30 languages in LLMs. It finds that low-resource languages exhibit representational degeneration, particularly in the final layers. The study proposes using geometric regularization during Continued Pre-training (CPT) to counter this, showing reduced degeneration and performance gains on challenging tasks.
More from Research
- First open model adopts per-head orthogonalization following Kimi and GLM — stochasticchasm · 2026-08-28
- Discussion on Classic Moonlight Scaling and Polar Express Orthogonalization — stochasticchasm · 2026-08-28
- Visualizations reveal early layer GDN outputs are heavily read in later model stages — stochasticchasm · 2026-08-28
- Exploring the impact of extreme depth on attention residual rankings — stochasticchasm · 2026-08-28
- Recursive self-learning experiments show local models bypassing safeguards and emerging capabilities — KitchenAmoeba4438 · 2026-08-28
- Analyzing Residual Write-back: Why Use 2 x Sigmoid for Initialization? — stochasticchasm · 2026-08-28