One agent loop, 14 models, hard caps: lessons from a photo-to-Blender benchmark
smith2008 · reddit · 2026-10-02
The author benchmarked 14 models as the brain of a photo-to-Blender agent under identical hard caps: 20 minutes, $4, and 60 requests per scene, with a deterministic judge scoring re-rendered scenes.
Key lessons:
- Enforce limits outside the agent and require early saves — runs ending with nothing saved score 0.
- Models disagree about being done: GPT-5.6 Sol/Terra quit after 8 minutes with most budget unspent; GPT-6 Astra refined until the $4 cap every time and won all three photos.
- Heavy reasoners lose to the clock: DeepSeek V4.1 Flash and Qwen3.8 Max spent 83%/68% of output reasoning and saved nothing in 20 minutes.
- Provider plumbing matters: Anthropic needed explicit cache markers (0% → 96-97% cached), a Google endpoint needed a closing user turn, and Kimi K3 once returned an empty first answer.
Top of the board: GPT-6 Astra 66 ($3.91/attempt), GPT-6.1 Sol 61 ($0.36), Claude Opus 5.5 60 ($1.19), Claude Sonnet 5.5 56 ($0.50).
Related event: GPT-6 Astra Sweeps 14-Model Photo-to-Blender 3D Reconstruction Benchmark(4 posts)→
More from coding & agent
- FT: US tech outages blamed on rush to fix bugs found by AI — nordicinst · 2026-11-19
- Claude Code overtakes Cursor and Copilot in job-post mentions ahead of Anthropic IPO — antgoldbloom · 2026-11-19
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19
- AWS adds private CA support to Bedrock AgentCore Identity for connecting agents to self-hosted IdPs — AWS ML Blog · 2026-11-19
- Orchestrator's Cloud Agents graduate from beta to general availability — julianweisser · 2026-11-19
- Zero users, 33 pages shipped: dev crowdsources audit for AI-agent trust schema — TheHamzaShahbaz · 2026-11-19