Before fine-tuning, distill traces into skills in-context to test if training is worth it
ziv_ravid · x · 2026-11-20
Ziv Ravid's thread (posts 7-8): before training you need a signal that improvement is possible. A way to check without training is distillation into skills — extract a set of skills from traces (manually or automatically), give them to an existing model in context, and rerun the eval. If that improves things, there's something in the data the model doesn't know and training makes sense; if not, he'd be less optimistic.
More from coding & agent
- NormGym benchmark shows agents learn from own norm violations but not from others — xuanalogue · 2026-11-20
- Every adopted 'compound engineering' company-wide so its codebase gets easier to maintain — every · 2026-11-20
- Start custom LLMs at 30B: runs on a single GPU and works with LoRA — ziv_ravid · 2026-11-20
- Not every agent task needs a general LLM — specialized small models by task type — ziv_ravid · 2026-11-20
- Fine-tuning is mostly iteration: cleaning data beats tuning hyperparameters — ziv_ravid · 2026-11-20
- Classification problems don't need a general LLM — use classifiers plus text encoders — ziv_ravid · 2026-11-20