Dev routes tool calls through a 421M model at 12ms to catch risky agent actions

looselyhuman · reddit · 2026-11-21

A developer building Pi's Auto mode uses a two-step inference check on tool calls: step 2 escalates to Qwen 3.5 4B (150ms per call, only for escalations), while step 1 needs a fast small model to flag risky calls (deletions, credential access, outbound transfers). Currently evaluating Laya 421M: given tool name + arguments, it flags effects and runs Q80 on a 3080 at 12ms per call. On a small playback set it caught 24/24 watched calls but false-flagged 4/6 benign reads, including a literal search for rm -rf. The author is soliciting comparable models.

Original post →

More from coding & agent

coding & agent channel →