How do you turn a real agent workflow into a useful evaluation set?
Super_Tough_4997 · reddit · 2026-10-24
A developer in TUM.ai's E-Lab asks how teams turn real agent workflows into evaluation sets: deciding what to test, designing first test cases, defining correct results and who judges them, and what real-world usage revealed that tests missed — including cases where the system, not the eval data, was at fault.
More from coding & agent
- Lee Rob's 45-minute masterclass explains how always-on AI agents work — Saboo_Shubham_ · 2026-10-24
- Prompt engineering is 'dead', long live prompt engineering: a cyclist-recipe prompt shows the craft persists — chrisalbon · 2026-10-24
- Few-shot prompting trick: turn examples into a list the system completes — mmitchell_ai · 2026-10-24
- Rewriting the world in Rust: nearly 1T tokens and 50k+ commits per month — doodlestein · 2026-10-24
- NeurIPS paper OAT: attribute agent failures by learning only from success flows — SharonYixuanLi · 2026-10-24
- Claude writes its own parallel-agent prompt and nails the brief, screenshot goes viral — hewarsaber · 2026-10-24