ProgramBench tests models rebuilding ffmpeg from scratch; top model resolves just 18.5%
jyangballin · x · 2026-10-24
Commenting on the new ./ProgramBench benchmark — 200 tasks and 248k+ tests asking LLMs to rebuild programs like ffmpeg from scratch — jyangballin predicts it will saturate in a fraction of the 3 years it took to saturate GitHub-issue-resolution benchmarks. Current leaderboard: Claude Opus 5.5 (xhigh) leads at 18.5% resolved / 70.5% almost at $37.30 per run, followed by Claude Fable 5.1 (9.0%), Muse Wonder Full (8.5%), GPT 6 Astra (5.0%).
More from Models
- Kimi K3.1 Pro debuts: beats Claude Opus 5.5 on Terminal-Bench 4.0 at ~1/3 the cost — Yuchenj_UW · 2026-10-24
- Reddit user on Kimi K3: pinnacle of intelligence, but Qwen wins for daily use — SrijSriv211 · 2026-10-24
- Nathan Lambert: Kimi K3.1 Pro shows Moonshot cracked RL, China keeps pace on capability — natolambert · 2026-10-24
- Omni cuts internal token spend 80% after switching workhorse model to DeepSeek Flash — josh_wills · 2026-10-24
- Inside the Artificial Analysis Cyber Index: three real-repo vuln-fixing evals, DeepSeek on top — ArtificialAnlys · 2026-10-24
- DeepSeek V4.1 Flash 1017 costs $0.42/task — 4x cheaper than GPT-6 Sol on Cyber Index — ArtificialAnlys · 2026-10-24