Tiny 1.5B local agent stops being confidently wrong with source-tier verification, finds real bug
UzairArain554 · reddit · 2026-09-06
The author ran an experiment on a CPU-only local setup (Qwen2.5-1.5B via llama.cpp): does teaching an agent to distinguish official docs from random blogs reduce confidently wrong answers?
Key findings
- The naive pipeline stored every source as "trusted" — all 7 answers got identical confidence labels regardless of source quality.
- The verified pipeline (source trust-tier classification + syntax checks on generated code) correctly distinguished sources 7/7.
- A real bug surfaced: a "confident" file-upload validator checking only extension and MIME type — both trivially spoofable — would have passed with full confidence.
- Format limitation: OpenAI-style JSON tool-calling failed 0/5 on the 1.5B model (llama.cpp only verifies native tool-calling at 7B+). Switching to a dead-simple text convention (SEARCH: <query>) worked 5/5.
Takeaway: small models can make the right call but can't always format it the "proper" way; simple text protocols are a viable substitute. Repo and full debugging log included.
More from coding & agent
- Researcher runs Codex + Astra on a plane, fixing papers and filling forms in parallel — soumitrashukla9 · 2026-09-06
- PLLDN: an open framework where LLMs assist but never decide your software stack — KeilerHirsch · 2026-09-06
- Heavy users say they'd pay $500/month for 50x Claude Code usage limits — GabGarrett · 2026-09-06
- 'Vibe coder' beats 'prompt engineer' in AI coding terminology debate — BLUECOW009 · 2026-09-06
- SkillGLoW: agents that remember procedures, not tasks, gain 17.2 points with 3.6x smaller memory — rohanpaul_ai · 2026-09-06
- dotey on Astra vs Codex: models can internalize how to use tools, never the harness itself — dotey · 2026-09-06