All tests passed, but Claude Code and a local 27B still changed how 2,672 real requests got judged
Stunning-Sherbet1853 · reddit · 2026-10-28
A team gave the same bug-fix task — a 4,915-line file with 191 regexes that judges Korean user requests — to Claude Code (Sonnet) and a local 27B model, grading on 36 written tests (19 shown, 17 hidden), 82 existing tests, and a replay of 2,672 real past requests against the fix actually merged.
Results (3 runs each):
- SWE-bench style, Sonnet passes 3/3; both models fixed the bugs (Sonnet 3/3, local 2/3)
- Both failed the same final check: the real-request replay. Sonnet judged 11-26 real requests differently per run; the local 27B, 8-113 (one zero was a harness bug)
Takeaway: all-green tests don't guarantee the behavior real users rely on hasn't quietly changed — the models only had the tests, while the merged fix was written with the 2,672-request replay in view. "Judged differently" isn't automatically "wrong", and one task × 3 runs can't rank models. The team is now rerunning with a tool that visualizes how a changed rule spreads across real requests without revealing expected answers.
More from coding & agent
- Unsloth adds autonomous browser use for local LLMs, Qwen3 27B demoed — danielhanchen · 2026-10-28
- C3 AI engineers break down why long-running agents fail: context rot, instruction positioning, cached tokens — C3_AI · 2026-10-28
- Agents now write code faster than humans can review — a Claude Code skill to triage dozens of PRs — KyeGomezB · 2026-10-28
- OpenAI: Codex weekly active users up 14x since start of 2026, France top 10 for API devs — reach_vb · 2026-10-28
- 8 open-source repos to self-host a lean AI agent stack instead of stacking $20/month SaaS — SaiTarunRonanki · 2026-10-28
- LangSmith now tracks agent cache hit rates to catch costly bugs — js_craft_hq · 2026-10-28