Engineer's long thread lays out a full playbook for enterprise post-training: evals first, real traces as data
ziv_ravid · x · 2026-11-19
Engineer zivravid published a long X thread systematizing how to run post-training (fine-tuning, distillation, RL) inside a company, ordered around four questions: why, how to measure, what data, which model.
- Why not just send everything to OpenAI: valid answers include cost, speed, data residency, or believing your own data wins on your task.
- Evals come first: he urges spending most of your time there — manually read real production task data, build intuition for good vs. bad cases, then define metrics.
- Data: the most useful asset is traces of real activity — user sessions, agent runs, tool calls, human edits to outputs. Labels can be sparse (did the user accept, edit, or reopen the ticket). Keep a small general-capabilities set, since task-specific training often breaks something else.
- Pre-training sanity check: distill skills out of the traces (manually or automatically), feed them to an existing model in-context, and rerun the eval to confirm the metric has large headroom before investing in training.
A practical, actionable roadmap for enterprise post-training.
Related event: Engineer's Long Thread Breaks Down Enterprise Post-Training(2 posts)→
More from coding & agent
- Human Principal launches private beta: cryptographic proof a human is behind your AI agent — yenkel · 2026-11-19
- GPT-6.2 builds code-only 3D fairy town and a deep-sea civilization in 10–20 minutes — OpenAIDevs · 2026-11-19
- Designers are being asked to do interior design while ignoring the architecture — round · 2026-11-19
- Dev builds a playable FPS in under 5 hours with GPT-6.2 and Unreal Engine 5.9 MCP — OpenAIDevs · 2026-11-19
- Stop vibe coding: a 7-step AI coding workflow from AGENTS.md to hooks — _jaydeepkarale · 2026-11-19
- You're using 10% of Postgres RLS: FORCE RLS, security_invoker and more — _jaydeepkarale · 2026-11-19