Why English knowledge edits fail in Chinese: 90% of the transfer gap is orthogonal residual

On the Geometric Bottlenecks of Cross-Lingual Knowledge Editing

Aastha A K Verma, Anwoy Chatterjee, Sparsh Jain, Tanmoy Chakraborty

NeurIPS 2026

cs.CL, cs.LG

2026-11-07

The cross-lingual edit gap has three geometric parts; ~90% is an orthogonal residual English-only edits cannot supply, and English residuals explain <4% of target variance.

What problem this solves

Multilingual LLMs carry a long-standing assumption: factual knowledge lives in an interlingua, a language-agnostic latent space, so what English learns Hindi can use. If that held for the weight space, a knowledge edit applied once in English would propagate to every language for free. It doesn't. On Llama-3.1-8B, ROME edits of CounterFact facts in English score 82.77 efficacy in English and 72.62 in German, but 49.61 in Hindi and 33.59 in Chinese. Earlier benchmarks treated this drop as an algorithm shortcoming. This IIT Delhi paper asks the structural question instead: is the failure baked into how the model organizes representations, beyond what any source-only editing algorithm can fix?

Method

The analysis targets ROME, MEMIT, and AlphaEdit because their weight updates have closed forms built entirely from source-language statistics: edit keys, edit residuals (the change vector an edit writes), and the key second-moment matrix C. Deterministic updates can be projected exactly onto target-language representations, which is what makes the dissection possible.

Transfer failure splits into two gates:

Two further questions sit on top of the decomposition: whether post-hoc correction from source statistics alone can work (no, there is a lower bound set by the private variance of target residuals), and whether pre-edit cross-lingual representation alignment can open the gate. Experiments cover Llama-3.1-8B and Gemma-3-4B, five languages (English, German, Hindi, Urdu, Chinese), multilingual zsRE and CounterFact, all on one H200.

Results

Efficacy on CounterFact, edited in English, evaluated per language (Llama-3.1-8B):

MethodEnglishGermanHindiUrduChinese
ROME82.7772.6249.6146.7033.59
MEMIT19.1213.027.098.650.87
AlphaEdit59.2989.6158.7454.0857.86

The key-space gate closes with typological distance: German's α peaks near 0.4 around layer 12, Hindi and Urdu stay below 0.08 at every layer, and distant languages compress keys into subspaces at principal angles above 75 degrees, nearly orthogonal. Batch MEMIT compounds the problem, with cross-fact interference ratios of 8-20 on Hindi and Urdu versus 4-5 on German. The value space is the dominant bottleneck: at the optimal editing layers (16-20, the semantic valley), the orthogonal component accounts for about 0.90 of total error, and a per-fact oracle that picks the best layer only brings it down to 0.77-0.80. Post-hoc correction hits a hard barrier: Ridge, MLP, and k-NN predictors explain no more than 3.6% of target-residual variance from English residuals, leaving about 96% unexplained.

Contrastive alignment (InfoNCE at layers 16-25 plus a language-modeling regularizer) opens the key-space gate dramatically: α for en-de rises from 0.212 to 0.858, en-ur from 0.015 to 0.517. The orthogonal mismatch barely moves (German ticks up from 0.470 to 0.486), and downstream results are mixed: Urdu improves 46.70 to 59.22 and Chinese 33.59 to 39.51, Hindi is flat, German falls from 72.62 to 61.17, and XNLI accuracy drops 4-8 points per language. Key-space alignment is necessary, not sufficient.

Why it matters

For anyone deploying multilingual models with editing-based fixes, the warning is direct: an edit that succeeds in English does not carry to other languages, and the paper explicitly flags safety corrections, where a fix effective in one language may fail in another. Multilingual deployments need per-language verification of edits.

For editing researchers, the paper delivers a reusable diagnostic vocabulary (α, orthogonal mismatch share), a negative result with a bound (source-only correction is capped), and a cleanly scoped positive result (alignment opens the key gate but cannot supply target-specific directions). The claims hold for closed-form updates, not for editing methods in general.

Limitations

The authors' own list: only three closed-form editors are analyzed; the cross-lingual key mapping is assumed roughly linear and downstream effects use first-order approximations; Proposition 3 gives only an upper bound, and Proposition 4 binds only corrections fed with source residuals alone, so methods using keys, prompts, or language identity escape it; two 4B-8B models and short factual answers are tested, with long-form generation unverified.

Three things stand out from the numbers. The correlation between the geometric gap and the behavioral penalty is modest (Spearman ρ between 0.114 and 0.221), so the geometry explains population trends, not individual outcomes. MEMIT scores only 19.12 in English under their batch setting, so part of its cross-lingual collapse is a weak baseline rather than geometry. AlphaEdit's CounterFact efficacy barely moves across languages (59.29 in English, 57.86 in Chinese), and the paper doesn't dig into why the same bottleneck weighs less on it. The per-language covariance statistics also come from machine-translated English Wikipedia, and the gap versus native-language text goes undiscussed.

Terms

Source

What people are saying

Related papers

All paper explainers