Claude Code's spring regression was a harness bug, not a model bug — a 12-agent harness comparison
Gooner-Psy · reddit · 2026-10-21
Prompted by Claude Code's measurable dip in March–April, the author revisits Anthropic's postmortem (via Fortune): one change made the model keep discarding its reasoning history mid-session, another capped responses at 25 words between tool calls — the API was unaffected. Same model, different harness.
He then compared published harness details from 12 teams (Codex, Claude Code, Cursor, Devin, Stripe's Minions, Manus, Agentforce, Shopify Sidekick, Sierra, Intercom Fin, Harvey, Hebbia):
- Claude Code is a thin layer around the model — tools, context management, file snapshots — with very little of the path fixed in code compared to Stripe or Salesforce;
- That works because coding mistakes are cheap to catch: compilers, tests, and git are decades of fast, verifiable guardrails that knowledge work lacks;
- The thin-harness risk is exactly what the regression showed: small harness changes move quality a lot, invisibly. The fix: rerun a fixed task set on every harness release and track reliability across repeated runs.
Two surprises: none of the 12 publishes cost per successful task, and 66 of 83 harness claims are self-reported. He also shares his own cost-measurement setup: headless Claude Code (claude -p) with all tools off, priced at API list price per accepted draft.
More from coding & agent
- Anthropic built Scion, a VCS just for coding agents — now open-source gitai replicates the design — JafarNajafov · 2026-10-21
- dsh-context: open-source plugin visualizes what's inside your agent's context window and where money goes — aigclink · 2026-10-21
- dsh-context: a visual dashboard for diagnosing agent context bloat, cache hits and token spend — aigclink · 2026-10-21
- NVIDIA expands Agent Toolkit with Knowledge Foundation, Kumo models and OpenShell runtime — NVIDIAAI · 2026-10-21
- Voice notes + AI auto-editing: a note workflow that finally stuck — Yobrogamer · 2026-10-21
- Five things healthcare teams must settle before letting an AI agent near patient data — alex_verem · 2026-10-21