Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory
Taeil Kim, Kangsan Kim, Sung Ju Hwang
cs.AI, cs.LG
2026-08-07
AMD distills a large teacher agent's successful trajectories into three hierarchical memories and injects them into 4B-8B students training-free; a Qwen3-4B student approaches GPT-5-mini on AppWorld, up 34.5 points.
Memory lets an agent reuse successful behavior and sidestep known pitfalls, and it works well on large proprietary models. Small models (4B-8B) hit a structural wall first: their task-success rate is low, so the trajectories they generate are mostly failures, and the memory they accumulate is dominated by unsuccessful experience with few usable successes.
A subtler problem is that even handing a large teacher's successful memories directly to a small student barely helps. The authors call this the capability gap. A teacher memory might say "log in first to play music," but the student cannot execute the login itself, and weak in-context learning means it cannot make sense of the teacher's notes even when they sit in plain view. It is the old "teacher too strong, student cannot keep up" failure of knowledge distillation, relocated to agent memory.
AMD leaves the student's weights untouched and instead gives it three memories of different granularity at inference time, all distilled from the teacher's successful trajectories:
A key design choice is that representation should match granularity: Workflow works best as prose, while Subtask and Function work best as code (replacing code with natural language drops AppWorld to 26.19%). Retrieval at k=1 is already optimal; larger k hurts because small models cannot absorb the extra noise. Function memory fires only on errors, so successful execution pays no context overhead.
Four students (Qwen3-4B/8B, Gemma4-E4B, Llama3.1-8B), all taught by GPT-5-mini. AMD beats zero-shot and three memory baselines (ReasoningBank, MemP, SASM), the last of which often underperforms zero-shot:
| Student | AppWorld zero-shot to AMD | BFCL V3 | ToolSandbox |
| Qwen3-4B | 14.88% to 49.40% (+34.52) | 15.50% to 38.50% | 16.28% to 20.16% |
| Gemma4-E4B | 24.40% to 54.17% | 37.25% to 46.00% | 18.22% to 21.71% |
| Qwen3-8B | 25.60% to 51.79% | 38.00% to 45.50% | 20.16% to 25.58% |
Average gains are 27.2, 11.2, and 3.4 points across the three benchmarks. Notably, Gemma4-E4B and Qwen3-8B surpass the teacher GPT-5-mini on AppWorld (54.17% and 51.79% versus 50.00%), so the student is reinstantiating transferable decision patterns rather than copying trajectories. Ablations show Subtask memory contributes the most (adding it to Workflow alone lifts Qwen3-4B another 25 points), while student-generated memory is near-useless. Gains peak at 4B, where the model is capable enough to use the memory yet still far from its ceiling, and diminish by 8B/14B. AMD also cuts interaction turns toward the teacher's level (Qwen3-4B on AppWorld: 23.8 to 14.9 turns, teacher 10.1).
For anyone wanting agent capability on a small model without training, AMD offers a reusable recipe: memories are built offline and read-only at inference, plug into four open models, and ship with code and a project page. It turns "the large model's experience" into a structured asset the small model can actually consume, and it puts the teacher-student compatibility question squarely on the table.
The authors concede three points: only text-based, Python-API or structured-function tool-use tasks were tested, leaving multimodal grounding and open-ended code generation unverified; memory is built offline and frozen at inference, so it cannot absorb the student's own online successes and failures or adapt to distribution shift; and a stronger teacher is not always better, since Qwen3-4B does better with GPT-5-mini than with the stronger GPT-5.5, leaving teacher-student compatibility unresolved. One more from reading the paper: all three retrieval paths rely on embedding similarity with a fixed threshold, and although the cross-split and self-excluded controls show the gains are not a task answering itself, both controls stay in-distribution, so cross-domain generalization is largely untested.