Jason Weston: Harness Optimization Hits 56.0, Distillation Adds 5.5 More
jaseweston · x · 2026-11-17
Jason Weston shares two experimental results for improving agent performance:
1. Harness Optimization (discover): a meta agent reads trajectories, finds where memory use breaks, and rewrites LOAD/OFFLOAD; the best candidates (5 runs each) become parents. Average test score: 56.0 optimized vs 44.6 best hand-written vs 36.1 generic.
2. Harness Distillation (demonstrate + distill): run the model with its optimized prompts, swap them back to generic while keeping its responses, then LoRA fine-tune. At test time with the generic prompt: 43.1 vs 37.6 for SFT on its own generic runs.
The idea is to bake the behavior induced by a good harness into the weights, so a plain prompt at inference still scores higher.
Related event: Jason Weston Boosts Agent Memory via Harness Optimization and Distillation(2 posts)→
More from coding & agent
- Stitch embeds AI agents in banking: the safety story is harnesses, not hack-proofing — alex_verem · 2026-11-17
- Agent Identity 101: one sequence diagram explains delegated, agent, and MCP auth patterns — adnan_hashmi · 2026-11-17
- How Comet improved a financial incident remediation agent's score from 0.34 to 0.57 with Opik — dl_weekly · 2026-11-17
- Meta's Self-Improving Memory: agents discover their own memory skills and distill them into weights — jaseweston · 2026-11-17
- Every shows Claude Fable 5.2 building interactive games — slow but token-hungry — every · 2026-11-17
- Two free Wan2.1 ComfyUI workflow sets: native nodes vs Kijai wrapper with speed boosts — asdfghmfker · 2026-11-17