Researcher challenges ProgramBench: top score is 4.5%, not 59% as pictured
srchvrs · x · 2026-10-13
@srchvrs publicly questioned the promotional figure for ProgramBench, a benchmark from Meta Superintelligence Labs, Stanford and Harvard: the image shows 59%, but the actual leaderboard's best fully-resolved rate is only 4.5% (Claude Opus 5 (xhigh) with mini-SWE-agent), suggesting a wrong picture or number.
ProgramBench asks agents to rebuild a complete codebase reproducing a program's behavior from only its compiled binary and documentation, across 200 tasks. Most frontier models score under 5% resolved (many at 0%), though the "almost solved" rate (passing ≥95% of behavioral tests) is notably higher.
More from Research
- Hugging Face researcher admits the decision index now faces MTEB's leaderboard-gaming problem — antoine_chaffin · 2026-10-13
- RT-SAFE targets real-time safety blind spot in embodied AI — repligate · 2026-10-13
- Researchers release research blog post months in the making — logangraham · 2026-10-13
- NeurIPS position paper warns generative molecular dynamics evaluation is misleading the field — FrankNoeBerlin · 2026-10-13
- Live talk: MinerU-Popo paper on fixing OCR's cross-page context failures — HamelHusain · 2026-10-13
- OpenAI math repo's 'exact complex arithmetic' claim draws fire over arbitrary-precision assumption — jessi_cata · 2026-10-13