Niche benchmarks expose huge gaps: best model scores just 5.5% rebuilding a codebase from a binary
Informal-Trouble2183 · reddit · 2026-09-06
A Reddit thread rounds up three 'deep capability' software engineering benchmarks where frontier models diverge far more than on mainstream leaderboards:
- Program-Bench: given only a compiled binary and docs, agents must build a full codebase reproducing the program's behavior—no decompilers, no internet. GPT-6 Astra scored just 5.5%, Fable 5.1 led at 7%, Kimi K3 2%, GLM 5.3 and GPT 5.6 Sol 1.5% each, Qwen3.8 27b and GPT 5.6 Luna 0%.
- SRE-Bench (vals.ai): understanding a real-world binary without source. GPT-6 Astra 88%, GPT-5.6 Sol 55.9%, Claude Opus 5 (max) only 12.5%.
- Code Migration (vals.ai): reimplementing working programs in another language. GPT-6 Astra 67.7%, Fable 5.1 54.6%, GLM 5.3 44.2%, Qwen3.8 27b 14.2%.
The author argues that binary-level comprehension is the real test of deep intelligence, and models diverge dramatically on it.
More from Models
- Meta researcher: horrifying infra, not talent, is why OSS can't catch closed frontier — A_K_Nain · 2026-10-23
- TypeSafe's Jev: a "Decision Model" LLM That Returns Floats at $0.042/M Input Tokens — ariG23498 · 2026-10-23
- Microsoft's MAI-Base-2 scaling ladder pegs the L42 run at ~1.8×10^23 FLOPs — teortaxesTex · 2026-10-23
- Local model agentic coding benchmark: 27B matches 125B, tool-calling is the lever — Joralk_ · 2026-10-23
- NVIDIA open-sources PersonaPlex-7B, a full-duplex voice AI model that listens and talks simultaneously — solyarisoftware · 2026-10-23
- NVIDIA's KumoRFM-2: a foundation model that predicts churn, fraud and demand straight from relational databases — solyarisoftware · 2026-10-23