ContinualSkillBench: Can LLM Agents Truly Evolve Their Capabilities?
Tianyi Guan, Yiding Wang, Haotong Yang, Siyuan Cao, Shirui Liu, Yi Hu, Jiaqi Li, Muhan Zhang
cs.AI, cs.CL, cs.LG
2026-08-05
Peking U.'s ContinualSkillBench: sequential tasks lift agents 16.9%, but a maintained skill library is no better than just keeping context; gains come from feedback adaptation.
Giving LLM agents a "skill library" is now standard practice. Tools like Claude Code and Codex let models write recurring operations into structured skill documents and pull them up at runtime. But those skills are mostly hand-authored and injected from the outside. In real deployment tasks keep arriving, and the open question is whether an agent can, simply by working through those tasks, distill its experience into reusable skills and get better on the ones that follow. Nobody had measured this systematically.
The traditional way to adapt a model to new tasks is fine-tuning the weights, which triggers catastrophic forgetting, where the model learns the new and forgets the old. So the field pivoted to in-context continual learning, leaning on long context windows to grow and retrieve a skill library without touching parameters. ContinualSkillBench asks whether that path actually works, and if it does, what is really doing the work.
ContinualSkillBench covers five domains: healthcare, law, mathematics, finance, office. Each has 100 interconnected subtasks anchored to three core skills, ordered by difficulty and skill dependency into an easy-to-hard chain so later tasks naturally depend on skills practiced earlier.
Task construction runs in three stages. From about 30,000 candidates drawn from OlympiadBench, LawBench, TAT-QA, GAIA, ClawBench and others, a model labels required skills, filters, and rates difficulty. Then 200 task pairs per domain get pairwise dependency judgments in both directions, forming a directed graph ordered with Kahn's algorithm. Finally humans review. The authors validate that the chain is real: 69.5% of eligible tasks reuse at least one core skill seen earlier.
Each subtask follows a three-turn protocol. The agent receives the task and its current skill repository, executes, then reflects on evaluator feedback and may use two meta-skills, Create Skill and Modify Skill, to add or revise skills. Evaluation uses four evaluator types (exact match and F1, numeric with tolerance, LLM rubric judgment, executable programmatic tests) and defines two rewards: raw reward averaged over all 100 tasks, and normalized reward computed only on tasks that produce valid output under both settings.
The key contrast is three execution modes: Independent (reset history and skill library every task), Sequential (work through all 100, retaining and updating the skill library throughout), and Pure ICL (same context and feedback, but no creating or modifying skills).
Three models were tested: GPT-4o, GPT-5.3-Codex, Claude 4.7 Opus.
Sequential execution lifted normalized reward in 14 of 15 model-by-domain combinations. Macro-averaged absolute gains were +0.071 raw and +0.078 normalized, or 16.2% and 16.9% relative.
| Model | normalized gain |
| GPT-5.3-Codex | +0.098 (largest of the three) |
| GPT-4o | +0.077 |
| Claude 4.7 Opus | +0.058 (strongest baseline, smallest gain) |
By domain, healthcare rose the most (+0.149), then finance (+0.076), law (+0.058), office (+0.054), mathematics (+0.052). The only regression was Opus 4.7 on mathematics (-0.008). In specific cells, GPT-5.3-Codex pushed healthcare exact match from 0.321 to 0.857 (+0.536) and finance F1 from 0.417 to 0.833, while GPT-4o went from 0.107 to 0.429 on healthcare.
The ICL comparison is the telling one. On law, finance, and healthcare with GPT-5.3-Codex, normalized reward came out at Independent 0.466, Pure ICL 0.605, Sequential 0.602. Maintaining an explicit skill library was, in aggregate, no better than just keeping the context and feedback. The split lives in the sub-metrics: explicit skills were sharper on exact match in law and finance and lifted healthcare programmatic tests from 0.250 to 0.500, while ICL scored higher on rubric judgments across all three.
The skill libraries themselves expose a model gap. GPT-5.3-Codex accumulated 205 skills across the five domains, compact but frequently invoked; GPT-4o accumulated 384, larger, less reused, lower quality, which the authors describe as fragmented and task-specific.
For anyone building agents, the headline takeaway is that the gains from running tasks in sequence come mostly from the model adapting to prior context and feedback, not from the explicit skill abstraction itself. That means heavy investment in skill maintenance and retrieval machinery may not pay off on average compared with simply feeding enough context and writing good feedback. This is an average, though. For tasks that hinge on reusable fixed procedures or demand precise outputs, explicit skills still earn their place.
The second point is a diagnostic. Weaker models tend to pile up bloated, fragmented skill libraries instead of consolidating them into transferable ones. Skill-library size and reuse rate are a mirror for whether a model is actually consolidating capability.
The authors flag that five domains is reasonable, but the tasks all come from a fixed set of benchmark sources, so long-tail cases, distribution shift, and messy real instructions are not covered. Because sequential evaluation is expensive in time and API cost, only three models and two harnesses (Codex CLI, Claude Code) were tested, with no Cursor, no Google CLI, and no exhaustive Claude/GPT/Gemini variants. The task streams are also curated and ordered, unlike real deployment, and the study covers only in-context skill evolution, not parametric continual learning.
Two things stay undertested. First, normalized reward is computed only on tasks valid under both settings, so part of Sequential's edge over Independent could be that it pushes more tasks into valid output and into the denominator, rather than doing better per task; the paper does not separate these. Second, the ICL-versus-Sequential near-tie rests on three domains and one model, so its generality needs more settings.