Replacing LLM routing calls with 15ms CPU encoders: workshop on deterministic routing
camerongreen95 · reddit · 2026-10-22
Almost every LLM app has a routing/classification step that starts as a JSON-schema prompt and becomes the slowest, most expensive, parse-error-prone part of the pipeline. A Oct 30 workshop led by Ben Auffarth proposes replacing it with:
- Bidirectional encoders (ModernBERT, LFM2.5 Encoder) for routing, sub-15ms inference on CPU
- SetFit few-shot training with just 8 examples per class
- Split conformal prediction so the router knows when to escalate to an LLM or human
- LangGraph workflows with approval gates and retry-safe state
- No JSON parse failures at the routing step
Runnable notebooks included; intermediate Python required, laptop or free Colab.
More from coding & agent
- Datadog team has been generating TLA+ via skills, eyeing a meta-skill for full suites — tianyin_xu · 2026-10-22
- Agent planned a whole trip solo: perfect logistics, zero vibe — tech__unicorn · 2026-10-22
- "AI swarms that patch the world's vulnerable software for free" — who's building this? — voooooogel · 2026-10-22
- Recursive launches Flint, a laptop coding model largely built by AI — TechCrunch AI · 2026-10-22
- Inside T800: Contrarian Reviews Handled by a Filter Agent Before Dispatch — lucasmeijer · 2026-10-22
- Local storage vs cloud: yaroslavvb argues security is the only real case as agents get cloud Macs — yaroslavvb · 2026-10-22