Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness
Pinzhen Chen, Koel Dutta Chowdhury, Xiaoya Xu, David Tan, Doreen Osmelak, Ona de Gibert, Ariun-Erdene Tumurchuluun, Ashok Urlana, Fedor Sizov, Hale Sirin, Jesujoba Alabi, Karrar Talib Abed, Mateusz Klimaszewski, Nikolay Bogoychev, Niyati Bafna, Patricia Schmidtova, Preksha Manjunath Shanbhag, Sherrie Shen, Vilem Zouhar, Vivek Iyer, Yasser Hamidullah, Yusser Al Ghussin, Zheng Zhao
cs.CL, cs.AI
2026-08-10
Localised FLORES across 27 locales, tested on 32 models: translation-tuned models drop most, two strong models may overfit FLORES, and models translate US content best regardless of language.
Machine translation benchmarks have long shipped with a default assumption: start from English, translate outward into hundreds of languages (FLORES, NTREX, and WMT24++ all work this way). Two problems follow. First, the English source is public, so models may have seen it in training and their scores are inflated by contamination. Second, in real use a Hindi-to-English translation usually starts from Hindi a native speaker actually wrote, not from an English news wire filtered through Hindi. Testing on English-translated text misaligns with how the system gets used.
A second, overlooked axis: one language spans many regions. Chinese in China, Singapore, and US diaspora communities looks different; Hindi lives in India but also in Germany and the US among migrants. Treating a language as a single cultural context erases the regional variation.
Cultivar targets both. It is a source-contrastive benchmark (ground one sentence in several locales and watch how scores move) and a locale-oriented one (assess several regional variants of the same language separately).
Cultivar is built on the FLORES dev set in three stages:
The result is 27 language-script-location combinations (21 languages, 8 scripts, 21 countries), 200 instances each. Over 98% of the data is newly localised; annotators accept about 70% of LLM output unchanged. Only any-to-English directions were localised.
The key design choice is a structure-preserving entity swap. Original and localised versions differ only in locale content, so any score gap isolates robustness to that content rather than translation skill. The paper defines d-BLEU as localised score minus original score: more negative means the model loses more ground on localised input. Crucially, this delta is near-independent of overall quality, because a weak model that scores poorly on both versions gets a delta close to 0.
Three findings emerge from 32 open-weight models (0.8B to 122B, runnable on two A100 80GB).
First, Cultivar and original FLORES produce highly consistent rankings: average Spearman rs at or above 0.98. Localisation does not reshuffle who is strong.
Second, rankings hold but absolute scores move. Most models land d-BLEU between -2 and +2, averaging -0.49. Two groups skew clearly negative. Translation-specialised models cluster in the negative region whether traditional MT (NLLB, MADLAD) or LLMs tuned for translation (the whole Hy-MT2 family at -2.06, -2.16, -2.28, plus Seed-X-PPO-7B). And the two strongest models collapse hardest:
| Model | Original BLEU | Localised BLEU | d-BLEU |
| Seed-X-PPO-7B | 53.18 | 45.64 | -9.68 |
| aya-expanse-8b | 50.51 | 44.50 | -8.87 |
Both beat Llama-3.3-70B and Qwen3.5-122B on FLORES, then fall sharply on Cultivar. A false-positive probe (checking whether removed original-FLORES terms leak back into translations of localised input) sits below 1% for almost every model, including these two. So there is no direct evidence of verbatim memorisation, but the disproportionate drop points to overfitting on FLORES domain, style, or sentence structure.
Third, bigger models weather localisation better. The delta rises with parameter count in 6 of 8 families, which suggests FLORES may understate the real multilingual ability of strong models.
Fourth, locales differ in difficulty. Hardest is cmnHansSingapore (d-BLEU -4.29), followed by Syrian Arabic, Japanese, and Mongolian; easiest are Malagasy (Madagascar) and Konkani (India). A controlled ablation on Hindi and Chinese, both localised into regions where they are not native, returns one consistent signal: regardless of source language, models translate US-grounded content best and do worst on Chinese grounded in China or Singapore. That is a US-centric bias in training data, and Cultivar gives a systematic way to surface it.
A manual error analysis on Chinese-to-English across three regions corroborates the picture. Named-entity errors make up over 80% of locale-specific errors and over half of all errors. Pinyin romanisation errors appear only in the China split (20% of its errors); number-and-unit errors cluster in the UK (22) and China (14) versus the US (5), covering jin, pounds, and century-to-year conversions.
For anyone building or evaluating translation systems, this is a cheap diagnostic. One contrastive run reveals whether a high-scoring model has real skill or has simply locked onto the texture of a public test set. Seed-X-PPO-7B posting a BLEU well above Qwen3.5-9B with a comparable error count is the kind of mismatch no single test set would catch.
For practitioners whose input is genuinely non-English-native (most real use), FLORES-style scores need a discount, and the locale-specific number matters more than the headline BLEU.
For teams training multilingual models, the counterintuitive result is that translation-tuned models are the most fragile on localised content. Optimising hard for translation may narrow the model's distribution toward public test-set patterns at the cost of cultural generalisation.
Cultivar covers only the into-English direction; English-to-X and X-to-Y remain open. Each locale was localised by a single annotator, so personal bias can colour whether a locale looks hard or easy. The FLORES-overfitting claim is indirect: no verbatim-memorisation evidence, inferred from score drops and the error-versus-BLEU mismatch, and the paper hedges with "potentially" and "hints". False-negative rates are an upper bound on localisation failure, not an exact error rate, since one concept can have several valid translations. The same-language ablation covers only Hindi and Chinese, a small sample.
One gap the paper does not fully close: it attributes the negative delta in translation-specialised models to distributional narrowing from tuning, but offers no training-data or ablation evidence, so the mechanism stays a plausible hypothesis.