Analysis of 220K Agent Tool Calls: LLM Judges Good at Tracking Progress, Weak at Catching Attacks

Hrishi Olickel of Southbridge.AI analyzed thousands of hours of agentic runs and about 220,000 tool calls to evaluate the LLM-based supervisor Jev, finding it the best method he has tested for measuring progress and completion, but unreliable for catching shirking and defending against attacks.

2026-09-21 ~ 2026-09-21 · 2 related posts

• 220K agent tool calls analyzed: model-watching-model Jev great at progress tracking, weak at security — hrishioa · 2026-09-21
• What thousands of hours of agent runs reveal about LLM-as-judge: good at progress, bad at safety — hrishioa · 2026-09-21