The Geometry of Low-Resource Language Representations
Francois Meyer, Jan Buys
cs.CL
2026-08-24
Nine LLMs and 30 languages show final-layer degeneration for low-resource text; CosReg only slightly beats vanilla CPT on IrokoBench above 4B.
The low-resource gap is usually blamed on missing data. Meyer and Buys at the University of Cape Town ask a more internal question: do the hidden spaces of decoder LLMs collapse in lockstep with how little text a language has online.
Two geometric failure modes show up again and again. Token vectors can point in almost the same direction, so pairwise cosine similarity is high and the space looks like a narrow cone. Or variance can sit on a few axes, so IsoScore is low and the model is not using the dimensions it paid for. Prior work mostly studied masked LMs and sentence similarity. Nobody had mapped this, at scale, for modern decoder models along a low- versus high-resource axis.
Measure first, then intervene.
They score 30 languages and 9 open base models. Languages are binned by character count in MADLAD-400, from English and Spanish (over 200 billion characters) down to Wolof and Lingala (under 100 million). The models are Llama 3.2 1B/3B, Llama 3.1 8B, Qwen 3 1.7B/4B/8B, and Gemma 3 1B/4B/12B. At each layer they sample FLORES tokens: 1,000 pairs for mean cosine similarity, 5,000 points for IsoScore. MADLAD is a public-web proxy, not the models' real mix.
The intervention is monolingual continued pretraining (CPT) on 10 African languages in WURA: one epoch, learning rate \(1\times10^{-5}\), sequence length 512, always on base checkpoints. Vanilla CPT minimizes the language-model loss only. Two regularizers, both applied to the final layer with \(\lambda=1\), are added on top:
They regularize the last layer because that is where the cross-lingual pattern is most consistent. Regularizing more layers did not help. Negative \(\lambda\) for I-STAR, which would encourage anisotropy, did not help CPT either. Evaluation is IrokoBench: AfriXNLI, AfriMMLU, and AfriMGSM, zero-shot and few-shot.
The geometric claim holds. Lower-resource languages have higher cosine similarity in most layers. The final layer is the only place where both metrics, on all nine models, line up with data size. Amharic is the outlier: Ge'ez script shares almost no subwords with the Latin-script set, and its geometry stays isolated from input embeddings onward. Gemma, the most multilingual family here, compresses the gap, but the correlation in deeper layers does not go away.
The regularizers move the quantities they target. CosReg drives final-layer cosine down hard (Qwen3 8B goes from \(-0.004\) under vanilla CPT to \(-0.691\)). I-STAR lifts IsoScore (about \(+0.027\) on Llama 3.1 8B). Each regularizer barely touches the other metric, and barely touches other layers. Vanilla CPT already lowers cosine a little, except on Gemma 3 12B, where cosine rises by \(+0.054\).
Downstream numbers are modest. Below 3B, every form of CPT adds a few tenths of a point at best, and regularization adds nothing. From 4B up, CosReg edges vanilla CPT by a thin margin:
| Model | Base avg | vanilla CPT | +CosReg |
| Gemma 3 4B | 24.2 | 25.9 (+1.7) | 26.3 (+2.1) |
| Llama 3.1 8B | 23.1 | 25.0 (+1.9) | 25.1 (+2.0) |
| Qwen3 8B | 25.0 | 25.2 (+0.2) | 25.3 (+0.3) |
| Gemma 3 12B | 33.0 | 36.3 (+3.3) | 36.4 (+3.4) |
The brighter cells are small: Llama 3.1 8B AfriXNLI 5-shot 35.3 to 36.1; Gemma 3 4B AfriXNLI 0-shot 36.1 to 36.9 and AfriMMLU 0-shot 31.9 to 32.7. On AfriMGSM, the hardest task, CosReg still beats vanilla CPT on larger models: Qwen3 8B 8-shot 9.0 to 9.6, Gemma 3 12B 0-shot 15.6 to 16.5. Absolute scores stay low.
CosReg beats I-STAR. Local separability moves the needle more than global isotropy.
Anyone doing low-resource CPT gets a cheap knob: no parallel data, one cosine penalty on the last layer. Geometry moves as advertised and other layers stay put.
Do not expect it to carry language adaptation on its own. Under 4B it does almost nothing. Above 4B the extra over vanilla CPT is 0.1 to 0.4 average points. This is a measurement paper that checked an intervention, not a new SOTA recipe. Reproduce the final-layer degeneration curve first; treat CosReg as an optional add-on.
The authors flag three limits. The analysis conditions on data size; Amharic's script and Gemma's architecture are noted and then left alone. They only intervene during CPT. Training from scratch and instruction tuning were not shown to work: African instruction data is thin, and instruction tuning itself failed to lift IrokoBench. Gains are inconsistent and small in absolute terms.
One extra caveat they underplay: on Qwen3 4B, vanilla CPT is \(-0.2\) versus the base model and CosReg is \(-0.3\). "Does not hurt" is not true on every checkpoint. Averaging three tasks and several shot settings also inflates relative percentages on AfriMGSM, where scores sit in the single digits. MADLAD is a public-web proxy, not the true pretraining mix.