UrduBench: human-in-the-loop translation pipeline brings standardized reasoning evals to Urdu
adnan_hashmi · x · 2026-11-03
A newly published ACL Anthology paper introduces UrduBench, a standardized reasoning benchmark for LLMs in Urdu.
- Method: to address machine-translation sensitivity and the lack of low-resource reasoning evals, the authors build a contextually ensembled translation pipeline with human-in-the-loop validation — ensembling IndicTrans2, Gemini 2.5 Pro, Qwen 3 30B and NLLB 600M, plus heuristic post-processing and GPT-based best-feature selection.
- Data: MGSM, MATH-500, CommonSenseQA and OpenBookQA are translated into Urdu, evaluated across five difficulty levels, multiple architectures, scaling settings and prompting strategies.
- Findings: multistep and symbolic reasoning is significantly harder in Urdu, and stable language alignment is a critical prerequisite for robust reasoning.
- Code and datasets will be released; the methodology generalizes to other low-resource languages.
More from Research
- Zyphra's ZHA brings subquadratic attention via hierarchical tree-structured indexer — yacineMTB · 2026-11-03
- GPT-6 edits Qwen weights directly, lifting MATH-500 from 7.9% to 48.1% with no gradients — KyeGomezB · 2026-11-03
- RSNA 2026 lab walks through fine-tuning a VLM for radiology end to end — PRouzrokh · 2026-11-03
- RSNA 2026 lab walks through fine-tuning a VLM for radiology end to end — PRouzrokh · 2026-11-03
- Trajectory launches post-training platform with GS1 and Continual Graders research previews — simonguozirui · 2026-11-03
- Wharton Study: Persuasion Principles Lift LLM Compliance from 33% to 72% — adrianscottcom · 2026-11-03