Simulated user drops agent success from 100% to 60%, UiPath open-sources dialog-mode eval harness
EvalRaccoonDev · reddit · 2026-09-23
UiPath argues most agent evals hand the model a perfect prompt, which is unrealistic. Their open-source eval harness codereval includes a simulated user that knows the requirements but only answers when asked.
Results across 5 tasks:
- 5/5 succeeded with the full prompt; only 3/5 with the simulated user, plus 44% higher cost.
- Failures came from the agent building the wrong thing because it never asked what was wanted.
- On an escalation task, both modes scored 5/5, but dialog mode cost +106% and +199% wall-clock time — meaning a single-prompt eval saw neither failure nor the time/money gap.
The harness and dialog-mode docs are open source.
More from coding & agent
- GBrain: plug your own memory, tools, and skills into any AI — garrytan · 2026-09-23
- Podcast: building a playable game with $8 of parts and AI assistance — aishashok14 · 2026-09-23
- Agent kept searching but never opened the source: four runs expose a hidden failure mode — memokris · 2026-09-23
- Probability-scored filtering with a small model beats LLM summarization for RAG and context compaction — marlene_zw · 2026-09-23
- TypeSafe classifies RAG passages with probability thresholds to fight noise and prompt injection — marlene_zw · 2026-09-23
- Dev: Opus 5.5 finds bugs that eluded other models for weeks, shows new agency — doodlestein · 2026-09-23