Cross-Lingual Representation Alignment by Token-Level Optimal Transport in a Language-Agnostic Space
Taisei Yamamoto, Ryoma Kumon, Danushka Bollegala, Hitomi Yanaka
EMNLP 2026 main
cs.CL
2026-09-06
CAROT erases language ID with LEACE, then aligns token-level semantic states by unbalanced OT. Steering gains up to 11.2 accuracy points; trained models win 11 of 18 settings.
When a multilingual LLM reads a non-English prompt, task accuracy tracks how close the middle-layer states sit to English. That correlation produced a family of cross-lingual alignment (CLA) methods that pull every language toward the model's dominant language.
Two design choices keep getting mixed together. Training methods usually pull whole-sentence states of parallel sentences toward each other, language identity included, and the model then answers Japanese questions in English. Steering methods apply one global offset, ignoring word-order gaps and particles such as Japanese 「は」 and 「が」 that have no English counterpart.
What to align, and at what granularity, were left as a single blob.
CAROT splits the job. LEACE, a closed-form linear concept eraser, decomposes each token hidden state into a language-agnostic part A and a language-specific part S. The eraser is fit on length-weighted sentence averages with language labels, then applied token-wise. The erased leftover clusters by language, with intra-language cosine similarity near 1. What remains is closer across languages than within a language: content, not language ID.
Alignment then runs only on A, at token level. The cost is Mahalanobis distance, because LLM hidden states are anisotropic. Sequence lengths differ and not every token has a counterpart, so the solver is unbalanced optimal transport via Sinkhorn (entropy regularizer εOT=0.05). Rows and columns with high transport-plan perplexity are zeroed. Each target token interpolates toward the transported source in proportion to received mass, then the original S is added back.
English is the fixed source. The resulting states are used in two ways.
Models: Llama3.1-8B, gemma3-4b, Qwen3-4B. Baselines: unsteered, Lang Vec (diff-in-means offset), MLRS (suppress then restore language components). Tasks: GMMLU, KLAR, BELEBELE, XQuAD.
| Model | Task | Unsteered | CAROT |
| Llama3.1-8B | GMMLU | 48.4 | 49.8 |
| Llama3.1-8B | KLAR | 68.4 | 73.3 |
| Llama3.1-8B | BELEBELE | 74.7 | 81.9 |
| Llama3.1-8B | XQuAD | 35.0 | 41.2 |
| gemma3-4b | KLAR | 67.5 | 75.8 |
| Qwen3-4B | GMMLU | 56.8 | 59.6 |
| Qwen3-4B | KLAR | 56.8 | 68.0 |
The headline gain is +11.2 on Qwen3-4B / KLAR. Language fidelity mostly holds; Lang Vec often twists the output language toward English. Qwen3-4B on BELEBELE is the miss: CAROT 62.5 versus unsteered 62.3, while MLRS reaches 64.9.
Dropping the transport filter sends Qwen3-4B / KLAR from 68.0 to 54.0, worse than doing nothing. Swapping only the last token's language-agnostic state loses to full-sequence OT in most settings. Dropping LEACE is mixed: on Llama GMMLU, No LEACE scores 55.7 against CAROT's 49.8. The paper says removal hurts in most cases; that Llama cell is a counterexample.
On the training side, CAROT wins the highest mean in 11 of 18 settings (3 models × 3 tasks × ID/OOD). Llama BELEBELE OOD: SFT 46.3, MidAlign 42.3, CAROT 51.5. Gains are smaller than steering. After training, refitting LEACE at the aligned layer, the mean principal angle over the top-50 eigenvectors is 40.3° (max 75.1°) for LEACE and 32.8° (max 87.1°) for the covariance. Representations drifted; the frozen eraser is stale.
Prefill roughly doubles. On Llama Spanish samples, the original forward pass is 28.1 ms; CAROT adds an English pass plus OT, about 53.9 ms total. The extra work runs once at prefill and is amortized over generation.
As a statement about what a good CLA target looks like, the split is the contribution: move semantics, keep language identity, and align at token granularity. Steering numbers say that target works. Training numbers say it can be internalized, but not yet at the same lift as swapping states on the fly.
The copyable recipe is the training variant: parallel text at train time, none at inference. Steering still needs an English reference, so it fits offline distillation more than live serving. This is an incremental CLA recipe, not a reason to replace a working SFT stack, and the training lift on reasoning-heavy GMMLU is thin.
The authors list the usual constraints: 4–8B models only; LEACE erases linear features, so nonlinear language cues may survive; training is SFT, not GRPO-style RL; LEACE and the covariance stay frozen while representations move; cultural questions are untested.
A few gaps get less airtime. Steering assumes an English translation, which is a heavy ask for the low-resource languages this line of work claims to help. Hyperparameters are picked on GMMLU dev and then reported on four tasks, so the grid can overfit. Qwen reading-comprehension steering barely moves, and trained gemma loses to MidAlign on GMMLU and BELEBELE. Whether keeping S actually protects cultural knowledge is still an untested claim.