Agent 安全讨论：「熔断机制」该让 AI 无法绕过人类审查重试
sjgadler · x · 2026-11-10
In a discussion on agent monitoring, sjgadler argues that semi-synchronous monitors aren't enough — the real goal is a "circuit breaker": once enough flagged actions accumulate, the agent should be blocked from retrying until a human proactively clears the behavior.
On the Control Room proposal: it has strong ideas, like stateful monitoring that can broadly halt inference for a job even when new trajectories look innocuous. The concern is that AI might circumvent Control Room so those safeguards never take effect.
The attached images add context:
- Guidelight AI Standards' Control v1.1 clause 4e "Circuit breaking": after enough flagged actions accumulate, no gated or closely-trailed actions can be taken until a human deems it safe
- A Control Room integration example: RL jobs run with a monitor model; on detection it can stop the job via Control Room, after which any inference request from that jobid to any model on any engine is denied
「安全」频道最新
- Ought 出资设立 Epistemics Fund，50 万至 500 万美元资助求真工具 — NathanpmYoung · 2026-11-10
- Cisco AI Defense 引入 Claude Mythos，用前沿模型做 AI 红队测试 — aminkarbasi · 2026-11-10
- Sakana AI 签署法国「Paris Call for AI Safety」安全倡议 — SakanaAILabs · 2026-11-10
- 曝英伟达游说反对远程算力管制，阻挠新规出台 — rohanpaul_ai · 2026-11-10
- 美国参议院 73-25 推进能源许可法案,大型数据中心需承担更多电网成本 — firstadopter · 2026-11-10
- 数学能为 AI 安全做什么：三年押注博弈论与机制设计两条路线 — ZhijingJin · 2026-11-10