Competence Gate: Using Small Model Confidence for Tool Calls
Synthium- · reddit · 2026-07-05
The author created a 10MB LoRA adapter and orchestration layer for Qwen3.5-4B that decides per query whether to answer directly, search the web, or retrieve local documents, refusing to fabricate facts if unverified. The core idea: small instruction models struggle to verbalize their confidence (tested 7 models from 3–9B, all hitting a confidence ceiling), but their internal activations actually contain this signal. The adapter reads this directly to gate tool calls. Experiments show this gating is better at catching errors than the base model's tool calls (d′ increased by 0.46, 95% CI [0.01,0.89]), and in multi-token cases, 87% were indeed wrong. A dual-signal version routes privacy queries to local retrieval, reducing the proportion of privacy questions sent to public web search from 22% to 10%. It runs locally on Apple Silicon/MLX and offers a GGUF version.
More from coding & agent
- Goal-driven AI needs verifiable success signals, or it invents its own — daniel_mac8 · 2026-09-11
- Is inference latency becoming the biggest bottleneck for production AI agents? — Euphoric_Sea632 · 2026-09-11
- Anthropic researcher: 99% of engineers now run swarms of 300+ self-improving agents — AlishaOutridge · 2026-09-11
- Gergely Orosz: Shipping 10x PRs With AI Agents, Sites Fill With Small Regressions — ducha_aiki · 2026-09-11
- Same Echo Maze prompt, three frontier models: all passed visually but shipped the same hidden bug — eyishazyer · 2026-09-11
- Astra storyboards plus Minimax H3 per-shot generation boost video success rates — Hailuo_AI · 2026-09-11