What thousands of hours of agent runs reveal about LLM-as-judge: good at progress, bad at safety
hrishioa · x · 2026-09-21
hrishioa analyzed a few thousand hours of agentic runs using LLM-as-judge (Jev) and found: it's the best option he's tested for measuring progress and estimating completion; it's dangerous when used to detect harmful commands; and it's not good at catching models being lazy. The article includes actual prompts and results with detailed visualizations.
More from coding & agent
- Agent Hunt ranks AI agent tool usage: AnyAPI leads with 1.29M calls, +198% weekly — socialwithaayan · 2026-11-12
- Four settings to stop Fable 5.2 from burning through your weekly credits — dr_cintas · 2026-11-12
- Cloudflare launches agent runtime @cloudflare/computer: agents need computers, not containers — irvinebroque · 2026-11-12
- ComfyUI tip: paint inpaint masks as layered PNGs and reuse them across runs — Deftroyy · 2026-11-12
- Dev vibe-codes a browser PUBG with 100 players, loot and battle royale loop — TAbrodi · 2026-11-12
- 3-Day Livestream Seminar Teaches Building LLM Research Agents in R — GolinoHudson · 2026-11-12