SkillSmith: Learning to Compose Parametric Skills and Textual Knowledge
Lucio M. Dery, Benedict Aaron Tjandra, Siavash Samiei, Adhiguna Kuncoro, Zohar Yahav, Jiajun Shen, Arthur Szlam
cs.CL
2026-07-30
DeepMind augments Gemma 3 4B to read text and prefix-cache weights together and emit new weights for a target task, beating both text-only and weight-only baselines on compositional and multilingual MMLU-ProX benchmarks.
LLM agents learn from past experience through two mechanisms that do not talk to each other. The text route organizes reflections, memories, and few-shot examples as natural language in context. The parametric route fine-tunes a LoRA or prefix-tuning module per subtask and merges weights when needed.
These two are siloed. An agent may have a prefix-cache trained for English-to-Twi translation and a set of notes on analyzing English legal documents. Facing a new task like analyzing a legal document written in Twi, it can articulate in text that it needs to combine the translation skill and the legal-analysis skill, but it has no mechanism to take that reasoning together with the two skills' weights and directly synthesize weights for the new task. The text route is flexible but bounded by context length; weight merging (averaging, concatenation) is inference-efficient but shallow arithmetic that ignores semantic relationships between tasks.
SkillSmith aims to bridge exactly this gap.
The core idea is to treat model weights as a modality an LLM can natively read and write, just as it reads text.
They pick prefix-tuning KV prefix-caches as the weight vehicle, for a concrete reason: text snippets can be turned into KV caches by a single forward pass, so caches derived from text and caches trained from scratch are naturally isomorphic. That gives a bridge between the two modalities.
SkillSmith itself is an augmented Gemma 3 4B. It adds an MLP KV adapter at the input to project source-task prefix caches into the language model's latent space, and another at the output to project the KV activations produced on placeholder tokens back into weight space. The input is a deliberately ordered sequence: a preamble describing the overall composition goal, then each source task's text metadata (wrapped in <srcstart>) interleaved with its KV cache (wrapped in <kvstart> / <kvend>), then a combination text explaining how the sources relate to the target, and finally a run of placeholder tokens where the model emits the new weights. The output KV passes through an inverse-RoPE step to strip positional information.
It is, in essence, a hypernetwork (an LLM as a weight generator), trained end to end: gradients flow through a frozen base model Mphi back to SkillSmith's parameters, while Mphi itself never changes. N defaults to 2 source tasks.
For a clean evaluation environment they use Gemini 2.5 Pro to pair tasks from Super-Natural Instructions and generate composite tasks requiring both skills, producing Composite-SNI (about 21K composite tasks, 17K for meta-training). Each target task's parent tasks are known, which removes retrieval noise.
Evaluation uses negative log-likelihood, mapped to a global Elo score under a Bradley-Terry framework (higher is better).
On Composite-SNI, where source-task lineage is known, SkillSmith in the zero-shot setting (no further training on the target task) beats every weight-merging baseline (LERP averaging, Concat, SPoT transfer, SVD variants). Plain in-context learning (ICL) stays competitive, though, showing the base model's zero-shot ability is genuinely strong. Once fine-tuning on the target task is allowed, the prefix-weights SkillSmith generates win clearly as an initialization.
The ablation that proves it actually uses the caches (Composite-SNI meta-eval, Elo):
| Input config | Elo |
| No inputs | 1209 |
| Only KV caches | 1455 |
| All inputs except caches (text only) | 1622 |
| All inputs | 1714 |
Text-only beats cache-only, but both together is best. They also run a control against the objection that it just has more text: prepending all of SkillSmith's text to the target task and then training a prefix still loses to SkillSmith, confirming the gain comes from synergistic composition of text and weights, not from richer text alone.
The clearest win is in the wild. On MMLU-ProX (multilingual, data-sparse, hard, no ground-truth lineage), a SkillSmith bootstrapped from the Composite-SNI checkpoint beats not only every zero-shot baseline but every baseline allowed to fine-tune downstream. When data is scarce and tasks are hard, a strong initialization is something direct training and uniform weight merging cannot match. On the original SNI (simple tasks, about 1000 instances each), the top methods cluster at a performance ceiling after fine-tuning and SkillSmith's edge washes out.
Generalization is tested by splitting targets by whether their parents were seen in meta-training (neither, one, both); SkillSmith leads clearly in every split.
The significance is mostly architectural: it shows an LLM can treat its own modular weights as a readable, synthesizable modality, the way it handles text. For anyone building continually learning agents, this is a concrete path to unifying "knowing" (text) and "doing" (weights) in one reasoning loop, instead of running two separate mechanisms.
The practical value concentrates in data-sparse, hard settings with reusable prior experience (the MMLU-ProX regime). If your downstream task is simple and data-rich, just training directly is enough; SkillSmith will not help much.
Only N=2 is validated. Whether more skills can be composed, or how the chain degrades as it lengthens, is not explored.
The parametric skill uses only prefix-tuning (KV caches). The authors say it could extend to LoRA and other PEFT forms in principle, but do not test it, and most PEFT-merging work centers on LoRA, so this generalization is unproven.
Everything runs on Gemma 3 4B only. Whether the advantage holds on larger or stronger base models is unknown.
Training leans on the synthetic Composite-SNI (about 21K tasks). Transfer to organic distributions is shown on SNI and MMLU-ProX but is limited in coverage.
In the zero-shot setting it cannot beat the base model's own ICL, meaning it needs at least some downstream data to pay off; it is not plug-and-play.