ProgramBench Eval: Claude 3 Opus Incurs a Staggering $50 Cost Per Task
jyangballin · x · 2026-08-13
ProgramBench, a joint effort by Meta Superintelligence Labs, Stanford, and Harvard, has released its full leaderboard, per-task costs, and trajectories.
Data reveals that running Claude 3 Opus is exceptionally expensive, hitting a record high compared to prior models: a single task costs $50.53, takes 257 turns, and outputs 357k tokens.
Related event: Claude Opus 5 Tops ProgramBench Coding Benchmark(7 posts)→
More from Models
- Unverified claim: Apple and Meta's 816B-parameter Muse Spark Open runs on two Mac Studios — TansuYegen · 2026-10-16
- Gemini 4 model card: model eliminated months of trial-and-error on bioweapons research — Puzzleheaded-King584 · 2026-10-16
- Every AI picks 7: Anthropic engineer's rule for escaping model taste convergence — rubenhassid · 2026-10-16
- Brain teaser: strip all attention, MLP and embedding matrices from a Llama 3 — how many trainable params remain? — Sauers_ · 2026-10-16
- Lean v4.35.0 ships verified kernels — one written and proven consistent by Claude — geoffreyirving · 2026-10-16
- One flagged seat froze an entire Claude team org — founders share backup plans — RexDouglass · 2026-10-16