Competence Gate: Using Small Model Confidence for Tool Calls
Synthium- · reddit · 2026-07-05
The author created a 10MB LoRA adapter and orchestration layer for Qwen3.5-4B that decides per query whether to answer directly, search the web, or retrieve local documents, refusing to fabricate facts if unverified. The core idea: small instruction models struggle to verbalize their confidence (tested 7 models from 3–9B, all hitting a confidence ceiling), but their internal activations actually contain this signal. The adapter reads this directly to gate tool calls. Experiments show this gating is better at catching errors than the base model's tool calls (d′ increased by 0.46, 95% CI [0.01,0.89]), and in multi-token cases, 87% were indeed wrong. A dual-signal version routes privacy queries to local retrieval, reducing the proportion of privacy questions sent to public web search from 22% to 10%. It runs locally on Apple Silicon/MLX and offers a GGUF version.
More from coding & agent
- A 9B Ollama agent can run a fully local DJ radio with tools, memory, and TTS — pinku1 · 2026-07-27
- Bugbot rejects an MCP permission flag because it would break path-scoped isolation — zeeg · 2026-07-27
- One GPT-5.6 agent is guarding a Blink security system while another makes a parody rap album — repligate · 2026-07-27
- An agent got unblocked by reusing a logged-in browser, not stealth tricks — armanidev_ · 2026-07-27
- Paper argues graph topology can become the core operating system for AI agents — theomitsa · 2026-07-27
- Claude Code desktop adds UI markup feedback for smoother visual editing — EricBuess · 2026-07-27