ProgramBench Tests LLMs on Rebuilding Software from Scratch; Top Model Solves Just 18.5%

ProgramBench challenges language models to rebuild complete programs like ffmpeg from scratch across 200 tasks with 248,000+ test cases; Claude Opus leads with only an 18.5% solve rate, though the author expects the benchmark to saturate far faster than earlier GitHub-issue benchmarks.

2026-10-24 ~ 2026-10-24 · 2 related posts

• ProgramBench tests models rebuilding ffmpeg from scratch; top model resolves just 18.5% — jyangballin · 2026-10-24
• ffmpeg-rebuilding benchmark will saturate in a fraction of the 3 years SWE-bench took — jyangballin · 2026-10-24