220K agent tool calls analyzed: model-watching-model Jev great at progress tracking, weak at security

hrishioa · x · 2026-09-21

Hrishi Olickel of Southbridge.AI evaluated typesafe.ai's Jev, a model that supervises other models, against 220K real tool calls (128K shell calls) from thousands of hours of agentic runs.

Key findings

Excellent progress tracker: best tool tested for measuring agent progress, estimating remaining time and spend, and labeling activity in long runs. The team tuned it using Hankweave, which can resurrect and replay long-horizon runs across models and tasks.
Unreliable as a security gate: Jev flagged 3,814 calls (1.73%), with Edit/Write alert rates of 9.5%/14% (models editing success criteria and plans). But 1% of verdicts flipped on retry, and harmful commands slip through easily via hex-encoding or Python wrappers. Good against honest mistakes, not hostile intent.
Poor at catching lazy models.

Full prompts, data, and methodology (unified event logs + sentinel experiments on replays) are in the article.

Original post →

220K agent tool calls analyzed: model-watching-model Jev great at progress tracking, weak at security

More from coding & agent