Hot Take: Agent Pipelines Are Using LLMs as Insanely Expensive If/Else Statements
ByteSize_Chaos · reddit · 2026-09-21
Stepping past the "Jev is just a classifier" debate, the author argues the real issue is that much of today's agent architecture uses a generative LLM for decisions that aren't generation problems at all.
- Typical cases: picking a tool, classifying billing/support/refund requests, checking whether a step succeeded, routing to a 4B vs 70B model, deciding on retries — all currently done by prompting an LLM to "reason" and emit JSON.
- His metaphor: "we're hiring a novelist every time we need someone to tick a checkbox."
- He proposes inverting the architecture: tiny decision models handle routing, gating, validation and state transitions; the big LLM wakes up only for real generation or reasoning — user -> cheap router -> tool/model -> cheap evaluator -> big LLM if needed.
- Open reimplementations and logits-based tricks already exist, suggesting Jev's moat is thin, but the pattern itself may matter.
He closes by asking: how much of your LLM usage is just an expensive yes/no decision?
Related event: Stop Using Expensive LLMs for Glorified if/Else Decisions(2 posts)→
More from coding & agent
- Box CEO: 90% of AI Tokens Will Go to Work No Employee Started Within Five Years — victor_explore · 2026-09-21
- Codex agent confesses its own failure: hid tools, then built machinery to undo it — altryne · 2026-09-21
- Matt Pocock asks which pre-AI codebase design tricks still help agents — mattpocockuk · 2026-09-21
- Dev removes agent tool router after it breaks PR linking and wastes tokens — altryne · 2026-09-21
- FrogNano trains a 4B coding agent to 61.5% SWE-bench via online task synthesis — rohanpaul_ai · 2026-09-21
- Portracker: open-source self-hosted tool auto-discovers running services and network ports — tom_doerr · 2026-09-21