The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents
Darshan Tank, Baran Nama
cs.AI
2026-07-25
Across ~5,800 agent runs, regressions cancel 59% of the gains skills add; the best libraries win by regressing less, and reliability hinges on grounding and verification, not procedure.
Bolting "skills" onto an LLM agent (writing a task's standard procedure into the system prompt so the agent follows it on similar problems) is a standard way to lift success rates. Evaluation usually reports the average gain. That single number conflates two opposite effects: some tasks the agent failed before and solves now (gains), and some it solved before and breaks now (regressions). A paper reporting only average success can hide a large set of tasks the skill itself broke.
The question here is concrete: how large is this "regression tax," why does it happen, and can it be avoided?
The core is a decomposition that splits pass-rate change into four paired outcomes: gain (failed without, passes with), regression (passed without, fails with), residual failure (fails both), and retained (passes both). Net effect equals gains minus regressions.
The experiment covers 5,832 runs: 486 tasks (94 OfficeQA-Pro questions over US Treasury documents plus 392 SpreadsheetBench Excel tasks) across four conditions (no skills, Anthropic library, OpenAI library, an in-house library) and three model-harness stacks (OpenCode with minimax-m2.7, Codex with gpt-5.4-mini, Claude Code with sonnet-4.6). Significance uses the exact paired McNemar test with a Bonferroni correction across 18 simultaneous comparisons.
By comparing the same task's trace with and without skills, the authors attribute regressions to three causes. Skill-description osmosis: the skill sits in context, is never invoked, yet changes behavior. Grounding displacement: the skill's procedure overrides the agent's correct reading of the input, so it queries the wrong table, range, or entity. Verification displacement: the procedure suppresses checks the agent would otherwise run on its output.
Regressions are a real cost. Across 18 conditions there were 553 gain transitions and 324 regression transitions; regressions cancel 59% of gross gains. The ratio is higher on OfficeQA-Pro at 66% (122 gains against 81 regressions), and 56% on SpreadsheetBench (431 against 243).
The more revealing result is who wins. On OfficeQA-Pro with OpenCode and minimax:
| Library | Pass rate | Gains | Regressions |
| Anthropic | 51.1% to 52.1% | 14 | 13 |
| OpenAI | 51.1% to 55.3% | 16 | 12 |
| In-house | 51.1% to 60.6% | 15 | 6 |
The three libraries post similar gain counts. The in-house library opens a 9.6-point lead mostly by cutting regressions from 12 or 13 down to 6. The best skill wins by breaking fewer tasks.
The cause distribution fits. Of the 81 OfficeQA-Pro regressions, grounding displacement accounts for 59 (72.8%), far ahead of osmosis at 14. Residual failures point the same way: nearly all return a number computed over the wrong quantity.
An underweighted trap is the grader. SpreadsheetBench's default value-only grader cannot evaluate Excel structured references or modern functions, so a correct formula the engine cannot recompute is scored as a failure. Re-grading with a full spreadsheet engine recovered dozens of tasks per condition (42 to 49 per library); of 663 failing tasks holding a formula, 226 (34%) were already correct. That chunk of "regression" is the grader, not the skill.
On significance, be honest: of 18 McNemar tests, only the three libraries under Claude Code with sonnet-4.6 on SpreadsheetBench survive Bonferroni correction (p<.001).
For anyone shipping agents, the takeaways are operational. Evaluate skills by decomposing gains and regressions; average lift alone overrates a heavy library and underrates one with side effects. Stop stacking more procedural "how-to" skills (the dominant ingredient of existing libraries) and reinforce grounding and verification instead, since those are where residual errors concentrate and where many regressions are recoverable. Before trusting an automated benchmark, get the grader right; that can be worth as much as fixing the skill.
The authors list several. Both benchmarks are office automation, where grounding and output format dominate, so the findings may not transfer to domains where the procedure itself is the bottleneck. Each stack couples a harness with its native model, so model and harness effects are not separated. Only three of 18 cells survive Bonferroni correction. Regression mechanisms were coded by one author with no inter-coder agreement, and the libraries are not length-matched, so library-versus-library contrasts are not a controlled token dose.
One more caveat: OfficeQA-Pro has only 94 items and SpreadsheetBench 392, so even with paired tests, point differences on a single benchmark should be read cautiously.