multiPL-e dataset contains hilarious regex errors, generating 'rsthon' tasks
code_star · x · 2026-08-17
Developer codestar highlights a glaring preprocessing error in the multiPL-e dataset's MBPP subset. The dataset attempted to convert Python to Rust code via a simple regex replace (swapping .py for .rs), resulting in absurd instructions to write a 'rsthon' function. Such rudimentary flaws, lingering in benchmark datasets, may have skewed evaluation results for numerous research papers relying on this data.
More from Research
- RLHF supersedes temp<1 heuristic, favors temp=1.0 — jessi_cata · 2026-08-17
- Why Simple Baselines Beat Fancy Biology Models: A Practical Guide — HongyiWang10 · 2026-08-17
- Multi-group IRT decomposes cross-lingual safety benchmarks — sanmikoyejo · 2026-08-17
- Nature reviewers flagged core flaw in 'science less disruptive' paper pre-publication — RexDouglass · 2026-08-17
- Opinion: Applying $100k of frontier compute beats decades of manual search — littmath · 2026-08-17
- Paper Maps Human-AI Creativity Across Codified, Tacit, and Novel Rules — sebkrier · 2026-08-17