Agent tweak is 15% faster at nearly same cost — the model admitted it overweighted raw token counts

dosco · x · 2026-11-18

Developer dosco shares a conversation where he challenges an AI's negative assessment: cache tokens are 90% cheaper and output tokens fell. The model then reverses itself — on AppWorld, the variant was 15% faster on average at essentially the same estimated cost (10/16 success), with per-task cost moving from $0.02900 to $0.02918 (+0.6%) at Luna's rates; output tokens dropped 23%, nearly offsetting extra cache writes.

The model notes the speedup came mostly from fewer checkpoint summaries, and that ToolSandbox got slightly slower and more expensive, so the gain is specific to the AppWorld sample so far. His takeaway: models hillclimb well but don't truly understand what they're doing — humans still bring the taste.

Original post →

More from coding & agent

coding & agent channel →