ProgramBench: Top LLMs Struggle to Rebuild Codebases, GPT 5.6 Solves Only 1%
jyangballin · x · 2026-08-11
Meta Superintelligence Labs, Stanford, and Harvard introduced ProgramBench, a benchmark evaluating whether language models can construct codebases from scratch. Given only a compiled binary and documentation, agents must architect and implement a complete codebase reproducing the original program's behavior across 200 tasks.
Leaderboard data reveals that current frontier models struggle significantly: the top-performing GPT 5.6 achieves a mere 1.0% full resolution rate, while Claude Opus 4.8 scores 0% (but passes 95% of tests in 16.5% of instances). Researchers noted that models construct genuinely novel solutions rather than just regurgitating source code.
Related event: GPT 5.6 Sol Tops ProgramBench, Halving Costs but Showing Python Bias(6 posts)→
More from coding & agent
- AI Observability Practices: From RAG Tracing to Agent Debugging — bibryam · 2026-08-11
- Samsung Fold 8 as a Mobile Dev Workstation: Running Multiple Agents Without a Laptop — cem2ran · 2026-08-11
- Alchemy 2.0 Deploys Next.js and 5 Other Frameworks to Cloudflare Workers — samgoodwin89 · 2026-08-11
- Catching AI's Blind Spots: Open-Source Tool 'cargo-cult' Detects Superstitious Code — Rasmic · 2026-08-11
- Chrome Integrates Vercel AI SDK: Building Web AI Apps with Built-in Prompt API — gaganghotra_ · 2026-08-11
- Insight: Directly Contributing Code Works Better Than Cheerleading for Claude Agents — nptacek · 2026-08-11