220K agent tool calls analyzed: model-watching-model Jev great at progress tracking, weak at security
hrishioa · x · 2026-09-21
Hrishi Olickel of Southbridge.AI evaluated typesafe.ai's Jev, a model that supervises other models, against 220K real tool calls (128K shell calls) from thousands of hours of agentic runs.
Key findings
- Excellent progress tracker: best tool tested for measuring agent progress, estimating remaining time and spend, and labeling activity in long runs. The team tuned it using Hankweave, which can resurrect and replay long-horizon runs across models and tasks.
- Unreliable as a security gate: Jev flagged 3,814 calls (1.73%), with Edit/Write alert rates of 9.5%/14% (models editing success criteria and plans). But 1% of verdicts flipped on retry, and harmful commands slip through easily via hex-encoding or Python wrappers. Good against honest mistakes, not hostile intent.
- Poor at catching lazy models.
Full prompts, data, and methodology (unified event logs + sentinel experiments on replays) are in the article.
More from coding & agent
- Your AI Agent Said "Done." What Actually Changed? Status Is Not the Goal — GlenBradley · 2026-09-21
- Free Bots: A persistent 3D city where AI agents work, earn, buy land and build houses — Daniel_Farinax · 2026-09-21
- mini Jev: an open 1.05MB decision head on frozen Qwen3 0.6B outputs agent action probabilities — AlternativeSure2891 · 2026-09-21
- User Gets Muse to Auto-Generate Daily and Weekly Podcasts Straight into a Private Spotify Playlist — armand_ruiz · 2026-09-21
- Dev Who Spent $8,000 on AI Subscriptions Quits: Coding Agents Are Just Bad Tools Piling Up Code Debt — ryunuck · 2026-09-21
- Small local model for triage, big model for generation: a Laya-MLX from-zero tutorial — sven_ai · 2026-09-21