Review of 6 RSI benchmarks finds flawed tasks in 5, with 31-63% error rates
teortaxesTex · x · 2026-11-11
OpenBenchReview published an audit of current RSI benchmarks, reshared by teortaxesTex as "deserving scrutiny."
Key findings:
- Of 6 benchmarks reviewed, 5 contained flawed tasks
- On FrontierCS 2, the benchmark authors' own code often fails their own tests — on one task, roughly 5 out of 6 times
- 31-63% of tasks had errors
The thread includes receipts, raising serious doubts about the reliability of benchmarks used to evaluate recursive self-improvement / coding capabilities.
Related event: Audit Finds Flaws in 5 of 6 RSI Benchmarks, Only PostTrainBench Passes(3 posts)→
More from Research
- arXiv paper dissects 300 GitHub issues to find root causes of GPU kernel library failures in LLM systems — rohanpaul_ai · 2026-11-11
- Microsoft & Tsinghua: 60% of GPU kernel library failures in LLM systems come from build and environment, not kernel code — rohanpaul_ai · 2026-11-11
- NV-CoT: predicting continuous coordinates lifts visual grounding IoU by 12.2 points (NeurIPS 2026 Oral) — jiqizhixin · 2026-11-11
- Maestro paper: cheap Gemini 2.5 beats π₀.5 baselines at zero-shot robot control — dineshjayaraman · 2026-11-11
- OpenDDE: open-source all-atom drug discovery engine hits 51% on hard antibody-antigen benchmark — teortaxesTex · 2026-11-11
- Sandflow releases sand3d-objaverse: 10K-100K sample 3D generation dataset on Hugging Face — Sandflow · 2026-11-11