ProgramBench goes from all-zero to Anthropic leading at 18% in months
jyangballin · x · 2026-10-27
- ProgramBench's author marvels at the progress: at launch every model scored a flat 0%, prompting the team to introduce weaker "almost resolved" metrics.
- The chart shows stepwise gains across labs: Anthropic's latest model hits 18% resolved, Meta 8.5%, and OpenAI 5%.
- Across releases from June to November, Anthropic has held a clear lead on the benchmark, with OpenAI and Meta trailing.
More from Models
- Blogger: Apple's AI will still be terrible even on iOS 270 — lipeng0820 · 2026-10-28
- SlopCodeBench October report: Astra beats Opus 5.5, xhigh effort deemed harmful — teropa · 2026-10-28
- OpenBMB ships 1.3B OCR model scoring 93.68 plus a 2B AI judge for OCR errors — Equivalent-Grass-527 · 2026-10-28
- Fangcun AI publishes third-party safety evaluation of DeepSeek V4.1 Flash-1017 for manipulation risks — pstAsiatech · 2026-10-28
- NVIDIA open-sources a LoRA post-training workflow to localize Nemotron with synthetic personas — NVIDIAAI · 2026-10-28
- Carbon-X hits frontier-level DNA understanding: Qwen3-14B finetune scores 66.7%, near GPT-6.1 — lvwerra · 2026-10-28