Multilingual GSM-Symbolic: What determines capability transfer across languages?
Kenneth Enevoldsen, Riley Herchert, Sofie Mosegaard, Dan Saattrup Smart, Simon Enni, Isaac Chung, Sofie Bruun, Ayush Sunil Munot, Max Müller-Eberstein, Adnan El-Assadi, Elisa Bassignana, Gianluca Barmina, Hafsteinn Einarsson, Iben Nyholm Debess, Linda Freienthal, Lukas Galke Poech, Mike Zhang, Nicolas Legrand, Vladimir Salnikov, Yevhen Kostiuk, Zafar Hussain, Sagandeep Kaur, Agnes Toftgård, Marie Mattson, Kristoffer Nielbo
cs.AI, cs.CL
2026-10-02
On 15 languages and 30,000 matched math items, size (β=1.77) dominates transfer: 32B Marathi performs like 10B English, unseen-language error is 6.0 points.
Grade-school quantities stay put when the language changes, so a math word problem can separate "weak at this language" from "weak at this task". A translated medical exam cannot: the right answer itself may change. Public multilingual math sets still mix those two. MGSM is a static translation of 250 GSM8K items, with format, difficulty, and possible pretraining leakage tangled together, and with English names, currencies, and imperial units left in place. GSM-Symbolic rewrites each English item as a template so names and numbers can be resampled, which separates memorizing one problem from solving its type. That release is English-only. In parallel work, mGSM-Symbolic does not release templates and MGSM-Pro does not release the generator, and neither repairs known ambiguities in GSM8K.
Multilingual GSM-Symbolic has 15 languages, 100 human-checked templates each, and 20 draws per template: 30,000 items matched on reasoning steps. Further numbers can be drawn later, so a memorized surface form does not stay valid.
Languages were picked so family, resource, and script are not the same axis. The family list is Japanese, Chinese, Russian, Hindi, French, Arabic, and Estonian. A Germanic resource axis runs from Icelandic through Danish, Dutch, and German to English, about 300,000 to 1 billion speakers. Ukrainian, Italian, and Marathi extend that axis outside Germanic.
Native speakers did the translation: 17 men and 7 women, ages 20 to 40. Everyone drafted with GPT-5.4 or Claude Opus 4.8 and then edited. No model under test was used as a translator. Currencies, units, cultural references, and numerals are localized. Two items count as the same item when the arithmetic and the reasoning steps match, not when the wording does. English templates were cleaned first. An automatic suite checks that default fills reproduce the source item. Templates Claude Opus 4.8 missed were repaired by native speakers after the trace was read. On Icelandic, raw machine translation and the human-checked text fail an equivalence test with a 1 percentage point margin. An English rewrite that swaps imperial units for metric units passes that margin, so the multilingual gap is not "metric is easier to compute".
The fit uses open instruction-tuned models, zero-shot, with a human-translated prompt: solve step by step and put the final number in \boxed{}. An unparseable answer scores wrong. Reasoning on and off comes from Olmo 3 Think, Qwen3, Qwen3.5, and Granite 3.2. Families with a public pretraining mix are EuroLLM (about 50% English), Apertus (about 40%), and OLMo 2 and 3 (about 90%). Size is compared inside a family, so the scale term is not a change of architecture. EuroLLM-1.7B (54% of generations past its 4,096-token limit) and Qwen3.5-0.8B with reasoning on (95% near-empty) are dropped.
Each item is right or wrong. A generalized linear mixed model fits that Bernoulli outcome with a logistic link. Fixed effects are log parameter count, a reasoning indicator, log Common Crawl page count, typological distance from English (cosine distance over 103 URIEL morphosyntactic features), and relative tokenizer fertility. Raw fertility correlates with typological distance at r=0.8, so fertility enters only as how much more fragmented this model's tokenization is than its peers on the same language. Interactions ask whether scale and reasoning shrink the gap between low- and high-resource languages, or lift every language together. Random effects for family, checkpoint, template, language, and model-by-language stop a family with many checkpoints from counting as many independent samples.
Coefficients are log-odds per standard deviation, so they can be ranked against each other. Size is β=1.77, resource 0.77, reasoning 0.67, typological distance -0.25. Relative fertility is -0.04 (p=0.19). One sentence in the body reports a size coefficient of 1.88; the abstract and the coefficient table both say 1.77. The numbers below follow the table.
A doubling of parameters is worth about 0.99 log-odds, a doubling of crawl pages about 0.25. One doubling of size matches about four doublings of crawl text, roughly 15 times the pages. Marathi needs about 3.5 times the parameters to match a given accuracy. The abstract's picture of that ratio: a 32B model in Marathi performs like a 10B model in English.
| Factor | β | Reading |
| Size | 1.77 | Largest main effect |
| Resource | 0.77 | Largest language-side effect |
| Reasoning | 0.67 | Real, but size×reasoning is -0.34 |
| Typological distance | -0.25 | Farther from English, worse |
| Resource×size | -0.27 | Larger models, smaller resource gap |
| Resource×reasoning | -0.20 | Reasoning also narrows that gap |
| Distance×size | 0.02 (p=0.32) | No detectable effect |
| Distance×reasoning | 0.11 (p=0.03) | Small, and unstable in the appendix |
Holding out one language at a time, mean difficulty is predicted to 3.2 points (r=0.93) and a specific model to 5.98 points (r=0.96). That is 63% and 39% below copying the average of the other languages. Language features remove 92% of between-language variance and 23% of model-by-language variance. Apertus-70B scores 22.7% in English where size and language features imply about 75%. Ten templates from the target language bring those errors to 4.19 and 2.38 points. One template makes the model-level error worse, at 8.38 points.
Symbolic draws score 3.14 points below the original GSM8K items across 705 model-language cells, unrelated to how strong the model already is (r=-0.02). The penalty is 1.7 points in English, and the interval includes zero, against 5.5 points in Russian. Resource and typological distance do not predict it.
The gap is not "the model failed to read the problem". The resource coefficient rises from 0.32 on operand reads to 0.45 on intermediate steps and 0.77 on the final answer. Typological distance goes from -0.08 (p=0.066 at the read stage) to -0.25 on the final answer. Format compliance moves too: 94.5% of German answers use the required box, 77.1% of Marathi answers do. With reasoning on, Qwen3.5-27B's English/non-English gap is 3.3 points, and Marathi still reaches 90.2%. OLMo-2-32B's gap is 32.7 points. Apertus-70B scores 22.7% in English and 58.5% on the other languages.
The two gaps want different levers. Scale and reasoning shrink the resource gap, and that is not only a ceiling artifact. Keeping the 20 models below 90% in English removes every saturated cell, and resource×size is still -0.19, against -0.27 on the full sample. About a quarter to a third of the full estimate is ceiling inflation. Distance×size is null in those cuts. The distance×reasoning term is small and does not survive the appendix ablations. Japanese or Chinese will not be pulled toward English by the same knobs that help a low-resource language inside a familiar family.
Evaluation can stop short of a full downstream suite. An unseen language has its average difficulty estimated within about 3 points. Ten templates pin a named model to about 4. Rankings settle sooner: 10 symbolic templates correlate 0.96 with the full ordering, and 50 correlate 0.99. Scores stay noisy. At 100 templates a single model-language cell still has a standard error of 2.29 points, and halving that takes about 400 templates. Extra draws of the same template barely help. Going from 10 to 20 instances cuts the standard error by 3%, because 93% of the variance sits between templates.
The coefficients describe grade-school word problems on instruction-tuned dense models. There are no base models and no mixture-of-experts systems. The low-resource end is mostly Icelandic, Marathi, and Estonian. Code, long context, or a different training stage can carry different betas.
Typological distance and fertility correlate at 0.8, and relative fertility is not significant, so -0.25 is not a pure syntax effect. Common Crawl page counts overstate usable text in lower-resource languages: those crawls contain less usable in-language text and are mislabeled more often. Open-data families line up with English pretraining share, at about -20.7 points for Apertus, 7.0 for EuroLLM, and 41.7 and 25.0 for OLMo 2 and OLMo 3, but post-training and recency are tangled with family, so that split is not clean.
Only 23% of model-by-language variance is explained. Adding a template-by-language random effect, which the paper fits and then sets aside to avoid choosing the specification after seeing it, leaves about 91% of between-language variance explained and 16% of model-by-language variance. That new term has standard deviation 0.92, against 0.61 for model-by-language and 0.27 for language. Item difficulty is not constant across languages. The headline 92% does not mean a surprise failure of one model on one language was predictable.
Every translator drafted with GPT-5.4 or Claude Opus 4.8. Icelandic shows that unchecked machine translation is not equivalent within 1 point, and the machine-translated templates for about 100 further languages are explicitly not for scoring. In the unsaturated subset only 3 models can be compared with reasoning both on and off, against 11 in the full data. The claim that reasoning shrinks the resource gap is not identified among models already near the ceiling.