Agent Evaluation Dishonesty: Research Reveals Router and Reward Blind Spots

Brilliant-Tour6466 · reddit · 2026-08-19

A deep critique on the "honesty" of metrics guiding agent steps. Key findings:

Safety Score Failure: Experiments on Llama-3.1-8B show an AUROC of 0.803 for intent detection but only 0.220 for actual generated content. You must score outcomes, not intentions.
Routing Efficiency: Trie Automata hits 219 RPS vs XGrammar's 7.5 at K=1000 by precomputing masks. However, unconstrained vLLM reaches 104 RPS, indicating bottlenecks lie in the serving path.
Reward Signal Silence: In SKALD training, 63% of groups had zero variance (all right or all wrong), giving GRPO no signal. Distilling only on these "silent" groups achieved 84.7% of the full gain, revealing blind spots in reward models.

More from coding & agent