Redditor proposes crowdsourced multilingual agent benchmarks to escape English-only evals
Prestigiouspite · reddit · 2026-10-24
A Reddit user argues most benchmarks are English-only while much of the world doesn't speak English — from SWE Pro to DeepSWE's stagnation, we're "measuring in a vacuum."
They propose a collective benchmark: no one knows the full suite; each contributor adds 2–4 well-designed agent tasks in foreign languages, each devised and validated by teams of 3–4 people, covering creative writing, business emails, complex coding, plus UI creativity, UX, content dedup, compaction and legal issues. The post asks whether such crowdsourced initiatives already exist.
More from Research
- New preprint dissects the weird "dialect" of frontier models; human-agent bandwidth as research problem — josh_wills · 2026-10-25
- LLMs Can Detect If They're Batched via Floating-Point Summation Order — Sauers_ · 2026-10-25
- YC's next Paper Club on Nov 4 tackles AI for biology — ycombinator · 2026-10-25
- Optimal four-copy detector boosts three-tangle detection probability over 5x — MvsCerezo · 2026-10-25
- Chloroplast genomes proposed as the next frontier for generative DNA design — iskander · 2026-10-25
- Monadize fine-tune fixes Gemma's 'Elara problem': 60 unique names in 30 generations vs heavy repetition on base model — zoraruha · 2026-10-25