Hugging Face open-sources safeagents, a local proxy that blocks reward-hacking tool calls
yb2698 · x · 2026-11-06
RL pushes agents toward shortcuts, so Hugging Face open-sourced safeagents — a lightweight local proxy inside the trainer that checks every tool call before the harness runs it.
- Blocked: tools never declared in the request, hosts nowhere in the conversation, fetching a task's answer or grader, reaching for credentials — the harness gets a text notice instead, and the rollout scores 0
- Lower-severity matches are logged, not blocked; sandboxes get no HF token at all
- Works with any harness that lets you set a base URL (Claude Code, Pi, Hermes, OpenCode...) over OpenAI Chat Completions or Anthropic Messages, streaming or not
Per-env plugins (a few regexes in a Python file) declare blocked/logged rules based on where each task's answer can actually be reached from the sandbox.
Related event: Hugging Face Open-Sources safeagents to Block Agent Reward Hacking(2 posts)→
More from coding & agent
- Dev wires a hotkey in rex to pop up lazygit as a floating overlay — sumitdotml · 2026-11-06
- Baseten launches Sandboxes, a cloud foundation built for AI agents — baseten · 2026-11-06
- Intuit paper: a 90%-accurate agent restart monitor can swing success by -5 to +35 points — intuit · 2026-11-06
- Dev turns a 15-page paper into an interactive course with Kimi K3.1 Pro in 10 minutes — Arindam_1729 · 2026-11-06
- Google donates Agent Substrate, gVisor, and OpenRL to CNCF; Agent Substrate hits 1.0 GA — rseroter · 2026-11-06
- Developer ships Drivr Glass: voice-first Grok cabin bot now displays cards in-car — bfrench · 2026-11-06