BenchCAD cost data: OpenAI internal model hits ~76% at ~$11/task, Astra tops out at 59%
jachiam0 · x · 2026-11-04
X user deredleritt3r published dollar-cost-per-task and score data for Astra and OpenAI's internal model on BenchCAD:
- Astra: low $0.44/task at 14%; medium $2.73 at 31%; high $5.11 at 41%; xhigh $6.67 at 45%; max $11.05 at 59%.
- OpenAI internal model (extrapolated): low $11/task at 76%; medium $23 at 83%; high $34 at 86%; xhigh $37 at 88%; max not shown.
Reblogger TimSchnabel argues the results reinforce warnings that models are approaching the ability to develop defense-relevant tech. The internal model is far more accurate but costs roughly an order of magnitude more per task.
Related event: Leaked Slides Show OpenAI Internal Model Beating GPT-6 Astra on BenchCAD(3 posts)→
More from Models
- Model-weights self-improvement demos show promise but lack evidence to beat fine-tuning — ccerrato147 · 2026-11-04
- Hype check: '20x better than RL' holds in 1 of 3 Atari games; math gain is just 44.6% to 48.1% — ccerrato147 · 2026-11-04
- AI hand-writes a Breakout network scoring 517; RL after 500M steps managed 25 — ccerrato147 · 2026-11-04
- MIT had GPT-6 Astra read Qwen's raw weights and fix a looping bug — no gradients, no training — ccerrato147 · 2026-11-04
- AI opens another AI's weights and fixes it without training: 10 facts from MIT's demo — ccerrato147 · 2026-11-04
- User asks why big labs won't let users choose what intelligence an LLM has — cyrus_zei · 2026-11-04