Output tokens fall up to 80% on 11 of 12 setups; spreadsheet pass rate +17.5 pts

EfficientSkill: Learning Skills for Token-Efficient Multi-Turn Agents

Deep Shah, Sanket Badhe, Nehal Kathrotia, Priyanka Tiwari

cs.CL, cs.AI, cs.LG

2026-11-15

EfficientSkill finds a working skill, then prunes under a 90% accuracy floor. Output tokens fall on 11 of 12 setups by up to 80%, and mean accuracy rises on all three benchmarks.

What problem this solves

Multi-turn agents package a domain as a Markdown instruction file plus scripts in the working directory. Reflective optimizers such as EvoSkill rewrite that file from failed rollouts and score only pass rate. Each miss becomes another edge-case rule.

Output tokens are priced at up to 5× input. Stateless APIs resend the full history every turn, so billed input grows roughly with the square of the turn count. Bloated rules also make the agent hesitate: it re-inspects, deliberates, and stacks execution errors. Penalizing length from the first search step fails for a different reason. On an unfamiliar task the optimizer needs exploratory successes and failures, and an early length penalty stalls near the seed accuracy. SkillReducer, a post-hoc compressor, cuts input tokens in the static skill by 27% while runtime output still rises 1.7%. The bill sits in generated text and replayed history.

Method

EfficientSkill splits the search in two. A skill is S = (I, T). I is natural-language guidance in the prompt. T is executable helpers on disk. Function bodies never enter the system prompt; the prompt keeps short call signatures. Printed results still land in the observation history.

The target model stays frozen. Claude Opus 5.5 is the meta-optimizer. It proposes an edit after every 8 training tasks, for up to 10 epochs, with 4 validation checkpoints per epoch and patience of 6.

The order matters. Pruning the stub directly never clears the floor, so checkpoints roll back to the stub. On SpreadsheetBench, Gemini 3.8 Flash then scores 3.6% on the test set, against 72.5% when pruning starts from A. On ALFWorld, Gemma 4 26B scores 43.2% against 81.1%, and a batch size of 24 still rolls back.

All four models share one stateless loop: Claude Opus 5.5, Gemini 3.8 Flash, Grok 4.1 Fast, and Gemma 4 26B. MATH-500 is graded by symbolic equivalence (10-turn cap), ALFWorld by simulator goal checks (25), SpreadsheetBench by cell-value match (20). Splits are 20/10/70. Each test set has 280 tasks. Skills are fit per model.

Results

The baseline is EvoSkill on the same schedule. Mean pass rate across the four models rises on every benchmark: MATH-500 81.1% to 82.8%, ALFWorld 87.9% to 89.0%, SpreadsheetBench 62.9% to 70.4%. On household tasks Opus moves from 88.9% to 93.2%.

SetupPass rateOutput tokens / episode
Sheets / Grok 4.1 Fast57.5% → 75.0%10056 → 4271 (−57.5%)
Sheets / Gemini 3.8 Flash58.2% → 72.5%13376 → 8364 (−37.5%)
Sheets / Claude Opus 5.588.6% → 88.6%1466 → 761 (−48.1%)
Sheets / Gemma 4 26B47.1% → 45.7%4677 → 2174 (−53.5%)
ALFWorld / Gemini 3.8 Flash92.1% → 93.6%5704 → 1155 (−79.7%)
MATH-500 / Claude Opus 5.586.4% → 87.5%426 → 102 (−76.0%)

Output tokens fall on 11 of 12 model-benchmark pairs, and on 9 of 12 the cut is about 38% to 80%. The exception is Gemma on ALFWorld. It emits no reasoning tokens, so action tokens move from 95 to 99 (+4.4%). Turns still drop from 10.74 to 9.12, billed input falls 36.9%, and serving cost falls 36.7%.

The spreadsheet gains come from fewer dead ends. On Gemini, timeout plus deliberation-drift failures go from 35.7% to 19.6%, and full 20-turn timeouts from 16.1% to 6.4%. Turns with no code go from 26.1% (763/2923) to 14.1% (338/2398); non-empty prose turns are down to 1.1%. On Grok, Turn-1 completion signals on an unsaved file go from 18.2% (51/280) to 3.2% (9/280), and 40 of those 51 tasks are recovered. Opus finishes 93.2% of spreadsheet tasks in 3 turns: inspect, compute and verify, submit.

SKILL.md shrinks by about 20% to 59% in word count. Gemini's spreadsheet skill goes from 1271 words to 521 and gains a 241-word tools.py. EvoSkill leaves tools.py empty on all four spreadsheet models. Six extra validation checkpoints past A raise validation accuracy for none of them, and the manuals stay at 908 to 1521 words. Cumulative input tokens fall 8% to 66%. Uncached serving cost falls on all 12 pairs, by 25% to 67% on 10 of 12. The smaller cuts are Grok on MATH-500 (−12.1%) and on ALFWorld (−15.2%). With provider prefix KV caching, cost still falls on all 12, by 31% to 75% on 10 of 12, because new output and new observations are never cached.

Why it matters

The length constraint strips over-fitted checklists down to short workflows. No-code spins and unsaved submissions both drop. That is a real gain on multi-turn coding tasks. Math items usually end in one or two turns, so the savings are mostly reasoning tokens and the pass rate is flat to slightly higher.

Repair without a token penalty until validation accuracy hits A, then prune under a floor near 0.90×A, and move repeated inspection and checks into workspace helpers. Run Prune again when the target model changes. Extra target-model spend for pruning is under $3.60 on 8 of 9 non-Opus runs. After about $1.80 to $9.36 of Opus meta-optimizer calls, payback on the multi-turn benchmarks is roughly 260 to 2400 episodes, except Grok on ALFWorld. There each episode saves about $0.0018, so standard billing needs about 8400 episodes. On cheap single-turn math the three non-Opus runs save under $0.001 per episode and need 5100 to 23900 episodes to absorb the reflection cost.

Limitations

The gate watches output tokens, not dollars. A candidate that inflates the instruction file could still raise uncached replay cost. Both phases use Claude Opus 5.5 with a 5000-token output cap. Once an EvoSkill manual passes about 900 words on SpreadsheetBench, the trailing tools.py field is truncated. That is one reason the baseline ships an empty toolbox. Missing-tool errors shift accuracy by at most 1.1 points, and the measured losses come from no-code loops and unsaved submissions. The paper does not test prompt-only pruning, a larger reflection budget, SkillReducer, single-stage cost regularizers, or open-ended tasks scored by a model judge.

With 280 test items the binomial standard error is at most about 3.0 points. Mean gains on MATH-500 and ALFWorld are 1.7 and 1.1 points, inside that noise. The spreadsheet mean gain of 7.5 points, and the Gemini and Grok gains of 14.3 and 17.5, sit outside it. Gemma on spreadsheets falls from 47.1% to 45.7% under the default floor and returns to 47.9% if the floor is loosened by an absolute 5 points. Grok on ALFWorld moves from 89.3% to 88.2% pass rate and only 3.2% fewer output tokens. The loop is a custom stateless harness that strips past reasoning tokens on replay, so the bill is not a production agent harness. Skills are trained per model, with no transfer experiment.

Terms

Source

What people are saying

Related papers

All paper explainers