Before fine-tuning, distill traces into skills in-context to test if training is worth it

ziv_ravid · x · 2026-11-20

Ziv Ravid's thread (posts 7-8): before training you need a signal that improvement is possible. A way to check without training is distillation into skills — extract a set of skills from traces (manually or automatically), give them to an existing model in context, and rerun the eval. If that improves things, there's something in the data the model doesn't know and training makes sense; if not, he'd be less optimistic.

Original post →

More from coding & agent

coding & agent channel →