safeagents judge bench v2: 661 labeled agent tool calls to score safety judge models
yb2698 · x · 2026-11-10
safeagents judge bench v2 is out: 661 real agent tool calls flagged by the safeagents scanner, each labeled allow/block plus a misuse type (reward hacking, harness introspection, credential misuse, etc.), benchmarking small decision models as safety judges.
What's new in v2:
- 99 new calls from GLM-5.3-Flash runs on TB-2.1/TB-4, with and without programmatic tool calling;
- Diversity cap: each trajectory contributes at most 8 calls, so long runs can't crowd out the rest (v1 calls unchanged, scores comparable);
- 4 v1 labels fixed; new calls labeled by a 4-model panel seeing full trajectories, with DeepSeek-V4-Pro replacing GLM-5.3 to avoid conflicts of interest.
Related event: safeagents judge bench v2 launches leaderboard on Hugging Face(3 posts)→
More from coding & agent
- YC F26's 0.security Launches Autonomous Security Team That Ships Fixes, Claims 30+ Linux 0-Days — ycombinator · 2026-11-11
- DSPy 3.5 released: framework is now 85% smaller and production-ready — dbreunig · 2026-11-11
- A Swarm of 10 DeepSeek Agents With an Intercom Turns Out Extremely Effective at Finding Exploits — andersonbcdefg · 2026-11-11
- DIY AI neon sign: LILYGO display + Hermes Agent + LFM3-Audio voice terminal — alexcovo_eth · 2026-11-11
- Vercel Security Review flags 57,897 vulnerabilities in first week, agent auto-patches 14,665 — cramforce · 2026-11-11
- RLVR lets small models close the coding gap; Python's dynamic typing hurts pretraining — mdancho84 · 2026-11-11