28KB activation probe detects unauthorized agent actions, beats text classifier with 0.814 AUROC
Think-Investment-557 · reddit · 2026-10-17
A developer tackled a real agent safety gap: "help me apply" means prepare a draft, not press Submit — yet agents routinely conflate the two, in email and beyond.
- Approach: froze six small models and trained a tiny logistic probe on their internal activations as they read the request, conversation, and pending action, separating approved actions from ones still needing permission.
- Results: MiniCPM5-1B worked best, with the useful signal in middle layers 13-14 rather than the last layer; the probe plus normalization data totals just 28,708 bytes.
- Evaluation: 0.8137 AUROC on a 3,000-example test set vs 0.7897 for a text classifier trained on the same 1,600 examples (paired difference +0.0240, 95% CI +0.0056 to +0.0429, p=0.0150).
- Notable finding: different models peak at different layers — there is no universal "read the last layer" rule.
- Intended as an extra signal alongside exact checks on what is sent, who receives it, whether the user saw the final version, and whether anything changed after approval.
Released on Hugging Face (mcpware/intentprobe-minicpm5-1b-authorization) and GitHub under IntentProbe; the author is soliciting clean authorization datasets and real overreach cases.
More from coding & agent
- Redditor Questions Muse, the #1 Downloaded App: Are AI Agents Ordering Uber Eats a Real Use Case? — eat-Razzmatazz1809 · 2026-10-17
- Ilya Razensteyn interviewed ~2 dozen engineers on coding agents; writeup due next week — thomasahle · 2026-10-17
- AIPOCH open-sources Open-Science: desktop app where AI agents run reproducible scientific analyses — rohanpaul_ai · 2026-10-17
- Stripe launches real-time Postgres sync for Stripe data, no webhooks needed — alexmacgregor__ · 2026-10-17
- MiMo-V2 paper: agentic groupwise grading redistributes RL advantages to curb reward hacks — Sauers_ · 2026-10-17
- Creator shares full AI music video pipeline: Claude for lyrics, Suno for song, local LTX for video — elonmusk · 2026-10-17