Test before you train: distill skills from traces into context and rerun your eval

ziv_ravid · x · 2026-11-19

A Hebrew X thread fragment on deciding whether to fine-tune: before training, look for signal that your measured task can improve a lot. One way to check without training is skill distillation — extract a set of skills from your traces (manually, automatically, or by inspecting the data), put them in an existing model's context, and rerun the eval. If it improves, your data contains something the model doesn't know and training is worth it; if not, that doesn't mean there's nothing there, but be less optimistic.

Related event: Before and After Training: Skill Distillation and Data Iteration(2 posts)→

Original post →

More from coding & agent

coding & agent channel →