Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents
Sourabrata Mukherjee, Kalika Bali, Sunayana Sitaram
COLM 26
cs.CL
2026-08-12
In 2.38M rollouts across 41 languages, frontier tool-using agents keep only 71% of their action policy when the task language changes, a gap final answers never reveal.
Ship a tool-using agent in more than one language and the question that matters is not whether it answers correctly in Hindi but whether it does the same thing in Hindi as in English: whether it retrieves before it computes, translates first, takes three steps or eight. For an agent the route is the product. A version that translates a passage before answering spends tokens and latency the English arm never spends, and a different route fails differently, since an error introduced by that translation step exists only on the non-English path and will never surface in an English regression test. Governance is route-level too: tool permissions, rate limits and audit trails are written against the sequence of actions a system is expected to take, so a policy validated on the English trace does not describe what happens in Hindi.
Standard multilingual evaluation is blind to all of this. Existing suites compare final answers, agentic benchmarks are almost entirely English, and the intermediate policy is discarded on both sides. Answer-level parity is compatible with any amount of route divergence, so the question is not measured at all today.
The authors make the action policy itself the measured object. Every model runs under one fixed ReAct scaffold (Thought: / Action: ToolName / Finish) with a shared five-tool symbolic alphabet; calls are parsed and compared but never executed, so the study targets induced tool-use policy rather than grounded execution. Scale is large: 8 models, 6 benchmarks, 41 languages, 809 language pairs, 2.38M rollouts, 505 cells. Each cell is run twice under identical conditions varying only the serving seed. Same-language agreement between the two replicates is Iwithin; cross-language agreement is Icross; both are cross-seed, so decoding noise enters identically and language is the only difference. Normalised retention is the ratio Icross / Iwithin, the share of a model's own reproducibility that survives a change of language.
The hard part is interpreting the traces. Naive averaged similarity gives a number that cannot be trusted, because five confounds sit between it and any defensible claim: no baseline (a model is not even self-consistent across two runs); a causal trace-length effect (6 to 7 pts); empty traces scoring 1.0; a ceiling capped by each model's own reproducibility (r=+0.97 at T=0); and a chance floor of c about 0.56, measured by permutation rather than assumed. Removing all five makes the effect larger, not smaller:
| Estimate | Iwithin | Icross | Corrected gap |
| Naive | 0.663 | 0.537 | +0.0625 |
| Matched, T=0.5 | 0.703 | 0.601 | +0.0861 |
| Matched, T=0 | 0.837 | 0.606 | +0.2074 |
The last row is positive in all 24 cells, every bootstrap interval excluding zero. Any story in which a confound manufactures the effect fails here, because every confound was suppressing it.
Under greedy decoding (T=0), four frontier models that share almost nothing converge on nearly the same value: each keeps 71 to 73% of its own action policy when the language changes. The four (Gemma-3-27B dense, Sarvam-M 24B Indic-specialised dense, Qwen3-235B-A22B MoE, and Llama-4-Maverick 128-expert MoE) span an order of magnitude in scale, two architectures and entirely different training mixtures. Model identity explains only 5.7% of the variance (versus 74.8% at T=0.5), and the across-model band shrinks from 13.6 points at T=0.5 to 2.6 at T=0.
Temperature matters because cross-lingual agreement barely moves as temperature rises while self-consistency falls sharply: sampling noise was masking the language effect, never producing it, and +0.0861 is a lower bound on the sampling-free +0.2074. The cost is that ranking models on the uncorrected absolute gap gives near-opposite orderings at the two temperatures (rank correlation -0.80); the paper states plainly that no per-model ranking from an uncorrected cross-lingual gap should be published.
The regularity breaks below roughly 10B parameters, and reads as a regime rather than a scaling law (n=2, flagged as a hypothesis). Ordering among small models is largely a chance-floor artifact: one 8B model has the shortest traces and the highest floor (two unrelated traces already score 0.669), and after correction it drops out of the band while a 4B model moves in, reversing which one looks better. The two models explicitly trained for multilinguality, Sarvam-M and Aya-Expanse, are the two that do worst at multilingual agentic behaviour, one on English advantage and one on scaffold adherence.
The mechanism is an English pivot. Translate is the most-used tool in every adapted benchmark, and the models' reasoning text is about 99% ASCII even on Devanagari, Tamil or Odia input: agents translate into English, plan in English, answer. This is causally testable. Removing the Translate tool lowers length-matched cross-lingual agreement in proportion to how much a model relies on it (Qwen3 spends 50.5% of tool calls on Translate, Sarvam 18.0%). A prediction registered before the compute was spent holds across four models: mandating Translate as the first action helps monotonically with each model's head-room. Told to reason in the task's language, models comply under 1% of the time (Gemma 0.79%, Sarvam 0.08%), so the pivot is not a preference prompting can switch off.
Finally, a single regex nearly manufactured a multilingual failure. GPT-OSS-120B yields no parseable trace on 76.4% of rollouts and under 2% accuracy, yet posts the two highest raw invariance scores in the study, because empty pairs score 1.0. The model is not broken; the parser is, writing "We will use Translate" in prose instead of the required syntax. Two worked exemplars raise its measured accuracy twenty-sixfold while its accuracy on readable outputs barely moves: the intervention made it legible, not smarter. The authors recommend reporting parse-failure rate beside every headline number and treating any model above about 20% as unranked (Aya-Expanse-8B, a fourth vendor, sits at 31.2%).
For anyone shipping agents, this lands on three things. First, "the answer is right" is not enough in a multilingual setting: nearly three tenths of a frontier model's action policy changes with the language, which means cost, latency and failure modes change with it, and English regression tests will not catch those paths. Second, the English pivot is structural and prompt-proof, so every non-English request pays a Translate tax that latency and cost engineering has to account for. Third, the measurement guidance is actionable: do not rank models on uncorrected cross-lingual gaps, and report parse-failure rates alongside every multilingual agent headline, because the single-regex trace extractors standard in ReAct harnesses can dress a parse failure up as a multilingual one.
The paper is candid about its limits. The regularity rests on four frontier models; per-cell there are 24 of them and the variance decomposition is done, but n=4 is still a bounded claim. The 71 to 73% figure holds only under greedy decoding and must always be quoted with its temperature. The sub-10B regime is an n=2 hypothesis, not a scaling law. Tools are parsed rather than executed, so the study measures induced policy, not behaviour under real environmental feedback, which may shift again in production. The fourth-vendor question is left open by Aya-Expanse's scaffold-adherence failure. The most counter-intuitive result, that multilingual-specialised models do worst, rests on one or two data points per conclusion and deserves dedicated follow-up.