ProgramBench: Agents Reverse-Engineer FFmpeg-Class Software from Binaries
nouhadziri · x · 2026-10-14
ProgramBench gives agents only a program's binary and docs — no source code. They must run it, figure out its behavior, and rebuild it from scratch (jq, SQLite, FFmpeg…).
It's hard: on FFmpeg, the best agent on the leaderboard passes only about 10% of tests, making the benchmark a strong probe of exploration and planning ability.
Related event: Study on scaling coding agent teams: placement matters more than headcount(14 posts)→
More from coding & agent
- Dev's honest scoreboard: pattern rules catch under 25% of prompt injections, guard model catches most — colinmcnamara · 2026-10-15
- Always-On Agents Emerge as a New Tech Layer: Dots, OpenDots and Pi Durable — rainerhahnekamp · 2026-10-15
- SpaceX's Cursor acquisition hailed as one of the fastest, best-executed integrations — kieranklaassen · 2026-10-15
- cantrell on Gemini 4 Argon: heavy thinker, building long decode to push past 64K token limit — cantrell · 2026-10-15
- Creator Shares Fix for AI Video Voice Consistency, Compares Claude vs ChatGPT vs Figma Make — Tegadesigns · 2026-10-15
- Indie dev builds AI game in 4 days: 297 Magnific assets, one-prompt Meshy animations — techhalla · 2026-10-15