SkillForge: Self-Distilling Agents for Project-Specific Issue Resolution
Silin Chen, Han Li, Xiaodong Gu, Yuling Shi, Haibing Guan
cs.SE, cs.AI
2026-08-19
SkillForge synthesizes issues from test-covered code and distills entity-grounded skills. SWE-bench Verified: DeepSeek-V3.2 66.4% to 72.2% Pass@1, GPT-5-mini 55.0% to 60.6%.
SWE agents now clear a useful fraction of real issues on cross-repo benchmarks. On one specific project they still rediscover the same brittle modules, API contracts, and implicit cross-file paths. The missing ingredient is project knowledge, not another generic scaffold.
Self-evolving agents take two reactive routes. History-driven methods distill from past issues, commits, or traces, and never see behaviors those traces never exercised. Online methods explore on the ticket in front of them, paying trajectory and token cost per bug. Both wait for an accident. Until then even a strong model is a generic repository tourist.
SkillForge mines project knowledge from the repo's own tests and code before any real ticket arrives. On a snapshot, it traces passing tests, finds the coordinated segments that implement one functionality, and asks an LLM to rewrite those segments under a strict mask: surrounding lines, location, indentation, and the test's goal, but not the original body. The rewrite exposes the gap between generic coding priors and project convention. Failed tests become SWE-bench-style synthetic issues, with a problem statement, a buggy patch, a reference patch, and tests.
Mini-SWE-Agent then solves those synthetic issues. Trajectories are distilled into a dual-level skill store. Global diagnostic skills, keyed by entity, record the entity's role in debugging, a playbook, and APIs that co-occur in repairs. Local intervention skills record how to edit that entity and which pitfalls to avoid, using both successful and failed traces. At test time, BM25 retrieves global diagnostics from the new issue text into the opening prompt. When the agent touches a file, matching local skills are injected as an extra observation. Retrieval is triggered by code entities, not by semantic similarity to old tickets.
Temporal isolation keeps the source commit of every synthetic issue before the target instance's gold commit. The Verified run synthesizes 577 issues. Temperature is 0, the action budget is 250 steps, BM25 takes the top 5 skills.
The scaffold is Mini-SWE-Agent. Backbones are DeepSeek-V3.2 and GPT-5-mini. Pass@1 is averaged over three runs. On SWE-bench Verified, SkillForge reaches 72.2% and 60.6%, +5.8 and +5.6 points over the bare agent. The strongest history method, MemGovern, is 69.2%/58.0%. The online method SWE-Debate is 68.2%/56.4% at $0.382/$0.167 per issue; SkillForge is $0.074/$0.066 after amortizing offline synthesis. A SWE-Smith single-function rewrite variant stalls at 68.0%/56.4%, and an LLM-summary variant at 68.7%/54.4%. The injection interface is not the gain; functionality-level, multi-segment synthesis is.
On SWE-bench Pro the scores are 34.1%/51.7% against a 28.3%/47.6% baseline, ahead of Live-SWE-agent at 32.4%/49.1%. Dropping global diagnostics costs 3.8/3.0 points; dropping local interventions costs 4.4/3.4. Cross-model transfer is diagonal. DeepSeek with its own skills is 72.2%, with GPT-5-mini skills 65.2%. GPT-5-mini with DeepSeek skills falls to 55.0%, matching the no-skill baseline. The store encodes how this model mishandles this repo, not a repo manual.
Across the seven largest repos SkillForge never regresses versus the baseline, with peaks of +13.6% on Sphinx for DeepSeek and +15.6% on Scikit-learn for GPT-5-mini. SWE-Exp regresses on Matplotlib and Astropy. Retrieval count and rewrite-segment count both peak near 5; dumping every skill or rewriting too many segments both hurt.
Project knowledge moves from "wait for incidents" to "distill from tests before launch." On a repo with a runnable suite, one offline synthesis pass is cheaper than per-ticket debate or live scaffold evolution, and closer to this codebase than GitHub-wide memories. Skills do not transfer across backbones, so a model swap means a re-distillation. For practitioners this is incremental and usable: leave the base model alone, add a repo-side memory.
Synthesis needs runnable tests; poorly covered regions contribute little. Problem statements are LLM-written from failure evidence and still differ from developer tickets. Skill quality is manually spot-checked, not verified at scale. Non-transfer across models multiplies offline cost. Backbones stop at DeepSeek-V3.2 and GPT-5-mini, not the strongest coding models of the period. Temporal isolation blocks future leakage; it does not block the softer leak that the test suite already writes project convention into the synthetic tasks.