Testing a local model as coding agent router: 0 of 48 decisions confident enough to switch
projectEscape · reddit · 2026-10-15
The author tried using a small local model (Llama-class open-source Laya) as a per-turn router for coding agents: cheap model for easy turns, expensive for hard ones. A local proxy sits between agent and provider; each new user turn Laya picks fast/balanced/strong/hold, doing nothing below 0.8 confidence.
Results were poor: across five prompt variants, agreement with hand labels peaked at 23/48, and zero decisions were confident enough to trigger a switch (best answerconfidence was 0.64). The router leaned to the middle tier (10 of 14 hard prompts got "balanced"); renaming options flipped 13/48 decisions; adding "hold if unsure" made it hold 44/48.
A telling side note: on 10 small gradable questions with thinking off, DeepSeek Flash scored 10/10 vs Pro's 8/10 and was faster — so "harder task → bigger model" is an assumption a router must earn. Next step: keep the router in shadow mode and validate against real outcomes (tests passing, retries, tokens) vs always-cheap and always-strong baselines.
More from coding & agent
- Claude Code Pushes Urgent Security Update Across All Installations — umeshonai · 2026-10-23
- Claude cut a 2.6-hour video export to 20 minutes — but only after being told to rebuild the server — byhewar · 2026-10-23
- 3 flight-booking rules my AI agent checks that I'd never do by hand — Efistoffeles · 2026-10-23
- OpenJet claims Qwen3.8-27B coding runs 1.9x faster than llama.cpp on one RTX 3090 — JeremyCMorgan · 2026-10-23
- Building an autonomous SEO agent in Hermes with three skills: programmatic pages, Keywords Everywhere, DataForSEO — EXM7777 · 2026-10-23
- Building a sports betting AI agent with onchain execution and human approval by default: architecture and lessons — Dry-Comfortable-2678 · 2026-10-23