Agent tweak is 15% faster at nearly same cost — the model admitted it overweighted raw token counts
dosco · x · 2026-11-18
Developer dosco shares a conversation where he challenges an AI's negative assessment: cache tokens are 90% cheaper and output tokens fell. The model then reverses itself — on AppWorld, the variant was 15% faster on average at essentially the same estimated cost (10/16 success), with per-task cost moving from $0.02900 to $0.02918 (+0.6%) at Luna's rates; output tokens dropped 23%, nearly offsetting extra cache writes.
The model notes the speedup came mostly from fewer checkpoint summaries, and that ToolSandbox got slightly slower and more expensive, so the gain is specific to the AppWorld sample so far. His takeaway: models hillclimb well but don't truly understand what they're doing — humans still bring the taste.
More from coding & agent
- Stitch launches AI agents natively in its banking OS, under bank permissions — CurieuxExplorer · 2026-11-18
- Seroter's Daily Reading: Levels of Agentic Engineering and Git's Future in the AI Era — rseroter · 2026-11-18
- Microsoft Ignite LAB663: a hands-on loop for improving agents with Copilot traces and Model Router — lee_stott · 2026-11-18
- "The model has no idea what its own biases are": designers on AI slop — petergyang · 2026-11-18
- Aware browser bets on feeds, not chat, as the right interface for AI agents — HaijunXia · 2026-11-18
- Ramen 1.0: open-source governed MCP platform with SSO, per-tool guardrails, and eval-gated deploys — Ok_Plum3595 · 2026-11-18