Skill Issue: Are Skills Language-Invariant in LLMs?
Bobby Cheng, Adam Gaber, Zhengyuan Liu, Catherine Arnett, Omer Goldman, Cheston Tan, Leshem Choshen
cs.CL, cs.AI, cs.GT, cs.LG
2026-08-26
518k self-play games across eight languages: English leads, Hebrew lags; Qwen's gap is ~2x Gemma's. English reasoning recovers 89.4% of Gemma's German TicTacToe deficit.
Multilingual eval has spent years showing that the same fact can be reachable in English and missing in another language. This paper asks a different question. Hold the model, the rules, the board, and the action space fixed, and change only the language of the interface. Does the model still play the same game?
If skills were language-invariant, two copies of the same model would split wins at random once you swap languages. Same-language matchups on the diagonal of their heatmaps are the control for that.
They extend TextArena so each player sees instructions and observations in a chosen language, while board symbols, coordinates, card ranks, numbers, and action tokens such as [bet] or [1] stay untranslated. Two copies of one model then self-play.
The run uses three similar-sized open-weight chat models: Gemma-4-E4B-it, Qwen3-4B, and Ministral3-3B-Instruct-2512. Eight languages were checked by native speakers: English, Arabic, German, Spanish, French, Hebrew, Malay, and Chinese. Six two-player games cover spatial play (TicTacToe, SimpleTak), solved combinatorial play (Nim), simultaneous allocation (Colonel Blotto), imperfect information (Kuhn Poker), and repeated social interaction (Iterated Prisoner's Dilemma).
Every language pair is played in both seatings, 400 games each way, at temperature 1.0. An invalid move gets one retry, then a loss. Across three models and six games that is 518,400 matches. The headline metric pools both seatings into a win-loss margin Δ = (wins − losses) / N, then averages a language against the other seven to get its strength μ.
They also ship translations of 65 games into 193 languages at several quality tiers. The numbers in the paper come only from the 6×8 human-verified slice.
English is the strongest interface on average; Hebrew is among the weakest. Gemma is the most even across languages. Qwen shows the steepest ranking. Chinese is not uniformly weak: for Qwen it outranks English on Nim and Blotto.
Language sensitivity is max μ minus min μ:
| Model | Blotto | Nim | TicTacToe | SimpleTak | IPD | Kuhn | Avg |
| Gemma | 0.44 | 0.02 | 0.47 | 0.43 | 0.25 | 0.05 | 0.28 |
| Qwen | 1.48 | 0.85 | 0.32 | 0.44 | 0.03 | 0.13 | 0.54 |
| Ministral | 1.28 | 0.49 | 0.36 | 0.28 | 0.74 | 0.20 | 0.56 |
Colonel Blotto moves the most (mean gap 1.07). Kuhn Poker moves the least (0.13). Qwen's 1.48 Blotto spread is large enough that a language swap looks like a different opponent.
How the models lose also changes. Gemma's English TicTacToe losses split 28.5% / 34.8% / 36.7% across rows, columns, and diagonals. Arabic shifts to 20.3% / 34.4% / 45.3%, with more diagonal collapses. SimpleTak repeats the pattern: English losses are 53.2% row versus 46.8% column; Arabic and Hebrew both sit at 42.8% / 57.2%. Boards are serialized row-wise, so columns and diagonals are harder. Weak interfaces make that bias worse.
In Kuhn Poker, Qwen's bluff rate with a Jack runs from 22.5% in Spanish to 47.4% in Arabic. Gemma is steadier on clearly weak or strong cards; the swing is the middle Queen, 41.6% bet in Malay versus 63.3% in Hebrew. Ministral's invalid-move rate rises from 9.3% in English to 20.9% in Hebrew, and folding a King against a bet lands between 10.4% and 23.7%.
Nim has a known solution: drive the Nim-sum to zero. Gemma barely notices language here. Win rates hug 50%, and the optimal first move is executed above 99% in every language (gap 0.02). Qwen does not. English wins 61.0% and plays the optimal first move 80.8% of the time; Hebrew 42.7% / 4.0%; Arabic 32.4% / 10.5%. Chinese is Qwen's best Nim interface at 69.8% / 74.3%. Ministral mentions the optimal strategy 538,093 times in English logs and 2,753 times in Hebrew. Of the remaining Arabic and Hebrew mentions, 70% and 50% come from the rare games where the model switched into Latin script (3.7% of Arabic logs, 1% of Hebrew logs). The strategy is in the weights. Weak interfaces often cannot fetch it.
Keeping a weak interface and changing only the thinking language recovers a lot in spatial games. Gemma's German TicTacToe sits at μ = −0.22; English reasoning lifts it to +0.20, 89.4% of the way to the English-interface ceiling. SimpleTak recovers 60.5% (−0.21 to +0.05). The same trick recovers only about 37%–49% in Kuhn Poker. Language hits different stages of the decision, so routing thought through English is not a universal fix.
Mean language strength correlates with Belebele (r = 0.71–0.79), 5-shot Global MMLU (r = 0.73–0.92), and FineWeb-2 web-text volume (mean r = 0.79; English estimated from FineWeb). Static scores predict who wins, not who is even. Gemma has the lowest Global MMLU mean (49.9) and the smallest language spread (0.28). Qwen has the highest mean (61.9) and nearly twice the spread (0.54). Malay beats Hebrew in every model despite a smaller FineWeb-2 corpus, which matches prior evidence that shared script helps transfer. Chinese has roughly 1/20 the web text of English; Qwen and Ministral still reach near-English margins, while Gemma's overall Chinese margin stays near zero. Data volume and script each explain a slice. The rest is model-specific.
If you ship a multilingual agent that has to follow rules, MMLU-after-translation is the wrong test. The same 4B checkpoint changes its risk profile, its spatial blind spots, and whether a known algorithm can be recalled, just from the interface language. English chain-of-thought is a useful patch for board games and a weak one for hidden-information poker.
The training takeaway is sharper. A higher average score does not mean more uniform behavior. Gemma looks worse on knowledge benches and more even at play. Covering Hebrew or Arabic is not only a data-quantity problem; you have to measure the policy.
This is a measurement paper, not a new trainer. Its value is turning "skills might depend on language" into a reproducible self-play setup, plus a 193-language environment dump.
The authors flag two issues. Training mixtures are closed, so they proxy data with public web counts; they tried Apertus, whose recipe is public, but invalid moves wrecked scoring. All three models sit at 3B–4B, which made half a million games affordable. Larger models may flatten the effect. Nobody showed that.
A few gaps sit outside that list. Action syntax stays in English brackets, so some invalid moves may be format failures rather than bad play. Translations started from Claude Opus 4.8 or GPT-5.2 and were then natively checked, which can still leave residue. Self-play measures relative language strength, not absolute skill against humans or a stronger model. Eight languages and six abstract games do not cover cultural knowledge or long-horizon tool use. Sampling at temperature 1.0 also spreads trajectories; the paper does not report a greedy rerun.