When Does Correction Become Repair? Mechanistic Auditing of Internal Interventions in Tool-Using LLMs
Jiayi Li, Ruizhe Li
cs.CL, cs.SE
2026-09-29
SAKIKO audits steering on tool-using LLMs: a +55 net-gain intervention corrupts 52 of 93 correct decisions it touches; one of seven models clears preregistered gates.
Tool-using agents have a failure mode that sits upstream of execution: before any tool runs, the model picks among four actions (call a tool, ask for clarification, answer directly, decline), and a wrong pick is a pre-execution error that good execution cannot fix. Recent activation steering work (ASA, CAST, Wu et al.) showed this decision can be pushed from internal representations, but evaluated it in binary action spaces, where leaving the wrong state lands you in the right one by construction. With four actions, exiting an error can dump the model into a third, equally wrong action, and the same push can corrupt decisions that were already correct. Aggregate net gain sees none of this. The question: when does an internal intervention count as repair rather than movement?
SAKIKO (Structured Adjudication of Keyed Interventions over K-way Outcomes) turns repair into a five-stage evidence pipeline:
The sealed protocol froze every vector, threshold, and dose before the test split was opened, with 59 budget-matched random directions as the null; the add-one Monte Carlo p-value floors at 1/60 ≈ 0.017.
The three sealed models were evaluated on the frozen cannotanswer → toolcall channel.
| Setting | Metric | Result | Control |
| Phi-3.5 · When2Call | net gain | +55 | random +14, sign-reversed +14, wrong layer +3, wrong channel −17 |
| Qwen2.5-7B · When2Call | net gain | +79 (92 fixed / 13 broken) | 10 random vectors mean +25.2, max +50 |
| Qwen3-8B · sealed | target gain | 0.276, CI [0.115, 0.425] | best of 59 randoms: 8 gold arrivals vs 38 |
| Qwen3-4B / Gemma-2-9B · sealed | target-hit point est. | 0.578 / 0.630 | CIs cross the 0.50 gate: DECLINE |
Direction specificity holds in five of seven models. On Mistral-7B the calibrated direction (+12) loses to sign reversal (+19) and the random mean (+29); on Llama-3.1-8B, 5 of 20 random vectors match or beat it. Steering looks like noise on those two.
Destination auditing is where the numbers split open. Phi-3.5's +55: of 200 routed errors, 107 reach the gold action and 46 land in other errors (target-hit 0.699); meanwhile 52 of the 93 correct decisions the Router touches break (55.9% against a 5% tolerance), and pushing the threshold to 0.9 does not help. On Qwen3-8B, activation steering and logit shifting post nearly identical net gains (+38 vs +35) with different destinations: lateral spillage of 14/52 vs 21/58 exits, target-hit 0.731 vs 0.638, while McNemar tests (p = 1.0 and 0.189) say the contrast is not significant. Bidirectional steering on MetaTool averages +8.6 and +13.0 over five runs.
Only Qwen3-8B earns an ADMIT (target-hit 0.731, CI [0.604, 0.846]; zero breaks, though only six correct decisions were exposed, leaving a one-sided break-rate bound of 0.393). Qwen3-4B and Gemma-2-9B carry point estimates of 0.578 and 0.630 whose intervals cross the 0.50 gate: DECLINE.
For anyone doing representation engineering on agents, this is an accounting standard, not a new steering method. Three directly usable lessons: aggregate metrics hide lateral errors and collateral damage, and two interventions with equal net gain can have different destination profiles; linear decodability does not imply steerability, as Router accuracy stays flat across mid layers while steering efficacy drops sharply; a favorable point estimate is not evidence, since resolving a 0.58 hit rate against a 0.50 null takes 245 exits versus about 30 for 0.73.
The authors' list is thorough. Formal adjudication rests on When2Call alone, with synthetic labels never re-annotated. The single ADMIT covers one channel of one model and supports no family-level or method-level claims. ASA and CAST were not re-implemented as baselines because the protocol was locked first. No setting certifies preservation on Router-exposed decisions; Qwen3-8B's zero breaks rest on six samples. The Qwen3.5-9B pre-intervention halt (29 reference rows against a threshold of 30) was split-sensitive, clearing in 56.7% of retrospective reallocations, and three procedural gate adjustments plus one mechanical restart are disclosed. One more from this reading: sealed-model target gains of 0.081 and 0.073 are thin even if they were statistically solid.