Agent Self-Improvement with Frozen Weights: The Added Value and Limits of Failure Traces
Arkid Mitra
cs.LG, cs.AI, cs.CL
2026-11-08
On a frozen Qwen3.5-9B, failure-trace notes beat a matched blind search by 4.0 points on service decisions, with no demonstrated gain in full chats or held-out tool use.
Small open agents often ship with frozen weights. What can still change is the harness: the prompt, notes, tools, and control logic around the model. A teacher can rewrite those notes from the agent's own traces. Scoring the rewrite only against the original harness mixes two gains. One is search: propose texts, keep the ones that score higher on a development set. That loop can raise accuracy without reading a single failure.
The failures' added value is whatever a matched blind search does not already get. Recent studies make the split necessary. Failure-derived experience rarely transfers to tasks the agent has not already seen.
The agent is Qwen3.5-9B, served with vLLM, built-in thinking off, weights never updated. Notes sit in one system-prompt slot. An empty slot is the benchmark's standard harness.
τ²-bench retail is a customer-service domain with a written policy, tested at two levels. Decision cases stop a synthetic conversation at one action and score it against a fixed answer: 2,997 sealed cases covering address, variant, and sequence changes. Conversation cases are the official benchmark, 114 tasks by 4 trials, pinned to v1.0.1 because later 2.x scores are not comparable. BFCL multi-turn is function calling in which the request and the tools already hold every needed fact, so asking the user cannot replace the right calls. Its 200 roots split into 50 discovery, 30 gate, and 120 held-out.
Lessons take the form 'WHEN situation THEN action'. On τ² the writer is the larger Qwen3.8-Flash-Next-FP8: propose, score on a validation split, keep or revert, 12 seeds per task. Both writers were given the target rules in the task description, so this test asks whether a 9B will follow a known rule, not whether anyone discovered it. On BFCL the writer is Qwen3.5-397B-A17B and reads 198 discovery episodes. The no-evidence search keeps the writer, loop, and budget. τ² replaces episodes with one fixed sentence; BFCL replaces them with the function documentation. A neutral BFCL note states no procedure and checks whether any added text changes behavior.
A separate gate screens single lessons. Candidates are switched on at random, and a hierarchical Bayesian model pools effects across contexts. Admission needs a credible gain above 1 point, a probability below 0.10 of losing more than 1 point in every context, and bounded harm. Primary rules were pre-registered. A blind BFCL audit reads items only, and the defect list was frozen before outcomes were opened. Official scores stay primary.
pass^1 is the single-trial task success rate. On decisions, the point estimate misses the 5-point gain registered as worthwhile.
| Setting | Contrast | Result |
| τ² decision accuracy | standard / blind search / trace notes | 70.3% / 76.8% / 80.9% |
| τ² decisions | trace notes − blind search | +4.0 [+2.3, +5.7] |
| vs standard harness | trace notes | +10.5 [+8.5, +12.5] |
| official τ² pass^1 | trace notes − standard | -1.3 [-5.9, +3.3] |
| BFCL held-out | gain of the admitted search lesson | -0.31 [-3.65, +3.12] |
| scorer defects | held-out / gate | 146/240, 42/60 |
Trace notes beat the blind search by 4.0 points. The superiority test passes, and the interval still includes gains larger than 5. Wrong writes, calls that are not exactly the correct one, fell by 3.1 points [-4.6, -1.7] relative to search. Of the +10.5 over the standard harness, search accounts for 6.5 points, about 60%, and that entire control gain sits on the task whose rule was already in the writer's instructions.
Full conversations show no demonstrated improvement. pass^1 moves from 82.9% to 81.6%. On 40 tasks never used in development the difference is -0.6 [-6.9, +6.3]. Conversations that edited items before the address fell from 6 to 0. τ² scores the final state, so the path can change while the score does not.
The τ² gate saw 20,400 decisions on 1,200 prefixes and admitted 2 of 12 lessons, with no held-out test. 'Change the address first' is +4.7 [+4.0, +5.4], 'list only the first change' is +2.7 [+2.0, +3.4], and together they are +7.4 [+6.3, +8.4], all 90% credible intervals. Four lessons that push the agent to ask or confirm again were rejected. The first admitted lesson restates, almost word for word, a rule the blind search also wrote.
Across 2,820 BFCL gate episodes, none of 8 trace lessons passed; estimates stayed within ±2.5 points. The lesson that passed came from search: 'when the directory is unknown, call pwd', at +4.9 [+1.9, +8.0]. On 18 items the audit called clean, that estimate is +3.9 [-0.4, +8.3] and includes zero. Over 5,760 held-out episodes the lesson's gain is -0.31, Holm-adjusted p = 1.000, and the gate's +4.9 lies outside the interval. Standard harness minus all eight trace lessons, run without the gate, is -0.52 [-4.38, +3.33].
A neutral note beats the standard harness by 4.5 points [+0.7, +8.4], more than either learned set. That comparison is descriptive, not multiplicity-corrected, and was not itself pre-registered. On clean items it shrinks to +3.4 [-1.9, +9.1]. The audit marks 146 of 240 held-out items (60.8%) and 42 of 60 gate items defective, κ = 0.84, mostly because a correct answer is scored wrong. On clean held-out items the lesson arms sit about 6 points lower: standard minus the search lesson is +6.0 [+0.2, +12.0], and minus the eight trace lessons is +6.2 [-0.5, +13.2], with Holm p of 0.149 and 0.153. The registered primary still uses the full official scores.
Without the blind search, the decision-level gain would be reported as +10.5 rather than +4.0, about 2.6 times as large. Those 4.0 points show that a teacher can phrase a known rule so a 9B follows it. The same libraries showed no demonstrated gain on full retail conversations or held-out tool use.
Repeated trials of published τ² retail setups flip 33% to 55% of task verdicts, so a few points are easy to lose in noise. The neutral note outscored both learned sets, so a new note has to beat a procedure-free note as well as an empty prompt. The notes do not travel. The procedure does: a matched no-evidence search, a neutral-note control, held-out confirmation of anything the gate admits, and a scorer audit frozen before the outcomes. Analysis code, the registry, and the defect lists are public; the experiment kits come from the author on request.
One frozen 9B with thinking off, two benchmarks, one teacher family. Decision items are snapshots: the agent cannot explore or undo a bad write. On τ² the task description already stated the target rules, so the decision result is not a test of discovery. On BFCL the search writer saw function documentation the trace writer did not, so the control does not only delete the failures.
The registered teacher was replaced before any result. The τ² gate was re-run after the first folder was lost; 0.27% of rows had reads answered from the original database, and dropping them leaves the admitted lessons unchanged. BFCL used temperature 0.7; the official decoder uses 0.001. A scorer serialization fix landed after the run, and verdicts that could be compared were byte-identical.
The two admitted retail lessons have no fresh-data test, and their gain is not attributable to the traces. About 60% of held-out BFCL items have scoring defects, and the clean subset points against the notes, short of significance after correction. The intervals still allow a few points either way on full conversations and held-out tool use. That is not a proof that the notes do nothing.