Silly regex error in multiPL-E dataset creates 'rsthon'
eliebakouch · x · 2026-08-17
An inspection of the multiPL-E dataset revealed a long-standing regex error during MBPP data cleaning where 'py' was replaced by 'rs'. This resulted in prompts asking to write 'rsthon' functions instead of Python, highlighting data quality issues in common benchmarks.
More from Research
- COLM 2026 Paper: Decoupling Factors Behind Multilingual Model Safety Degradation — davlanade · 2026-08-17
- Largest open experiment: frontier models close 82% gap to human record in autonomous AI research — inductionheads · 2026-08-17
- DeepMind demos model reasoning in blocks to self-correct — mtizard · 2026-08-17
- Benchmark reveals LLMs struggle with real-world tasks despite coding prowess — 数字生命卡兹克 · 2026-08-17
- Delip Rao Critiques AI for Science: Success Cases Suffer Survivor Bias — deliprao · 2026-08-17
- SCoPE Demo: 3D Video from Single Image Without Game Engine — yshan2u · 2026-08-17