Define Dimensions First: Hamel Husain's Method for Synthetic Eval Data
HamelHusain · x · 2026-09-26
Asked about the best approach for generating synthetic data, evals expert Hamel Husain gives a structured answer:
The common mistake is prompting an LLM to "give me test queries" with no structure, yielding generic, repetitive outputs. The right way is to define dimensions first — categories capturing different types of variation in user behavior — then combine them into queries, run them through the full system, and review traces.
- When to use it: to start error analysis before you have production traffic, or to test known failures that appear rarely in real data.
- Limits: synthetic data can't tell you how common a failure is in production and may miss domain-specific details; compare with real data as soon as it's available.
- Examples: a recipe app might use dietary restriction, cuisine, and query complexity; a support bot might use issue type, customer mood, and prior context. Start from failure hypotheses — if you lack intuition, use your app heavily or recruit friends.
More from coding & agent
- kaigen3d principles: code-first runtime and user-controlled frame loop make it AI-friendly — gdechichi · 2026-09-26
- kaigen3d engine demo: cross-platform AI-first 3D engine hits 330+ FPS on 6-year-old laptop GPU — gdechichi · 2026-09-26
- Data viz researcher turns chart-design research into open-source agent skill evident-charts — randal_olson · 2026-09-26
- Andrew Ng: rigid testing on early-stage AI projects is a recipe for stall — DeepLearningAI · 2026-09-26
- Peter Steiberger says he codes with Codex plus an OC harness, not Claude — steipete · 2026-09-26
- Can AI audit my 1000-video music library? A practical video-understanding use case — SkipMorrow · 2026-09-26