Routing ~50 tools for a local agent: three failed approaches and a GBNF fix
paulqq · reddit · 2026-09-09
The author built a hobby local-first agent in Rust (llama.cpp, runs on a 14B model, Obsidian vault as memory). The hardest part wasn't the model — it was deciding which of 50 tools the model may call each turn. Three phases, each wrong at first:
- grep/keyword matching: fails on synonyms, German chat, and "do that again".
- Routing phrases + embeddings: much better, but naive top-k with a hard GBNF grammar structurally locks the model into the router's single guess with no way out mid-turn.
- A small policy layer: when uncertain, widen to a cluster of related tools instead of committing to one guess.
Key lesson for small local models: feed the same offered-tools list to both the prompt and the GBNF grammar so they can never disagree on what's callable; when the router isn't confident, hand the grammar a small related set and let the model pick within the constraint. Full writeup includes Rust snippets.
More from coding & agent
- ComfyMax v0.2: a simpler frontend for local ComfyUI + MiniMax H3 workflow — DanielVeres · 2026-09-09
- AI connectors and browser use let perfectionists delegate without actually delegating — tushaarmehtaa · 2026-09-09
- Hackathon-winning memory system rebuilt as zero-dependency MCP server with SQLite knowledge graph — BoringCelebration405 · 2026-09-09
- Veteran dev claims a solo GTA-scale game in under 6 months is possible with UE plus AI tools — D3VAUX · 2026-09-09
- One good model beats a thousand sub-agents: a sharp take on agent architecture — tekbog · 2026-09-09
- Replacing a Telegram-to-Second-Brain summarizer workflow with Claude Code — sebpaquet · 2026-09-09