Turning agent traces into training data: capture, sampling, and labelling, worked through

spilldahill · reddit · 2026-09-04

Using a refund agent as a worked example, this post breaks down how to turn production traces into a training dataset:

Capture: every run's trace records the goal, each model call, tool invocations, observations, and outcome; the industry is converging on OpenTelemetry GenAI standard span attributes. Capture must be automatic and total.
Sampling: don't label everything. Three strategies—random (honest baseline), stratified (coverage of segments you care about), failure-weighted (highest signal per run). For 50k weekly runs, pull 500: a random slice plus all error/low-satisfaction runs, on a weekly cadence.
Labelling: read your agent's codebase first to derive an explicit checklist (refund within terms, no invented terms, escalate disputes, etc.) and score each sampled run against it—"feels fine" labelling falls apart at ten thousand runs.

(The post promotes the author's Overmind SDK, but the methodology stands on its own.)

More from coding & agent