4 models, one two-file bug: 3/4 passed, 10x cost spread, and the cheapest run was the failure

lulzxdxdxd · reddit · 2026-10-08

A follow-up to the author's earlier single-file test: four models fixed a bug spanning two files (a serializer and a TTL'd key-value store) and had to return complete corrected sources as structured JSON — the shape an agent actually needs.

Results: 3/4 passed all 5 hidden tests

|---|---|---|---|---|

| claude-haiku-5.5 | 0.10/0.50 | 5/5 | 18s | $0.00210 |

| gpt-5.6-luna | 0.20/1.20 | 5/5 | 43s | $0.00405 |

| claude-sonnet-5.5 | 2.00/10.00 | 5/5 | 10s | $0.01853 |

| gemini-3.5-flash-lite | 0.30/2.50 | 4/5 | 2s | $0.00181 |

The only failure was the cheapest run, gemini-3.5-flash-lite, on the cross-module part: escaping the delimiter its own output format uses — exactly what the two-file split tests.
Same deliverable, 10x cost spread ($0.00181–$0.01853), driven by the price list rather than effort: the cheap end writes at least as many tokens.
The author is explicit about what not to conclude: one trial per model, task still small (real work needs 5–10 files of context and a wrong spec), and passing hidden tests isn't a good diff.
The actual workflow change: pick models per step — cheap for "here's the file and failing test, make it pass," expensive only for "work out which files to touch at all." That's where the money is.

Original post →

4 models, one two-file bug: 3/4 passed, 10x cost spread, and the cheapest run was the failure

More from coding & agent