Test before you train: distill skills from traces into context and rerun your eval
ziv_ravid · x · 2026-11-19
A Hebrew X thread fragment on deciding whether to fine-tune: before training, look for signal that your measured task can improve a lot. One way to check without training is skill distillation — extract a set of skills from your traces (manually, automatically, or by inspecting the data), put them in an existing model's context, and rerun the eval. If it improves, your data contains something the model doesn't know and training is worth it; if not, that doesn't mean there's nothing there, but be less optimistic.
Related event: Before and After Training: Skill Distillation and Data Iteration(2 posts)→
More from coding & agent
- Stop vibe coding: a 7-step AI coding workflow from AGENTS.md to hooks — _jaydeepkarale · 2026-11-19
- You're using 10% of Postgres RLS: FORCE RLS, security_invoker and more — _jaydeepkarale · 2026-11-19
- Why you should always build your own LLM proxy: auth, caching, cost visibility — viksit · 2026-11-19
- Practical advice on fine-tuning small models: start around 30B, cheap LoRA experiments on one GPU — ziv_ravid · 2026-11-19
- Don't default to LLMs: classification models plus text encoders are the hot approach — ziv_ravid · 2026-11-19
- Engineer's long thread lays out a full playbook for enterprise post-training: evals first, real traces as data — ziv_ravid · 2026-11-19