A data-agent team says hardcoded capabilities made its LLM worse at complex queries
Tired40s · reddit · 2026-07-23
A team built an LLM data agent for Excel-style questions and found that a simple ReAct loop in V1 handled complex queries better than a heavily decomposed V2.
What went wrong in V2
- They split the system into many hardcoded capabilities such as rowcount, columnlist, and filteredranking.
- Each capability had its own parsing, retry, and escalation logic, plus stacked review loops.
- The result was a patchwork of narrow rules that did not share state, so modules re-asked users for information already confirmed elsewhere.
- Some deterministic modules also silently produced wrong computations, and the guardrails only checked grounding, not logic correctness.
The takeaway
The authors think the real failure was removing the model’s feedback loop: once the decision tree is pre-baked, complexity stops scaling and only accumulates.
Current redesign
They are moving back toward:
- a small set of general tools: schema lookup, query, calculate, ask-user
- a mandatory self-verification step before the final answer
They also ask for real-world patterns on structured params vs free-form SQL, and for techniques that catch logic errors, not just hallucinations.
More from coding & agent
- Spring AI 2.1.0-RC1 adds tool-call confirmation, pausing agent loops before real-data access — therealdanvega · 2026-10-26
- 250 Builders Hack With Codex at Berlin Community Hackathon Ahead of OpenAI DevDay — paw_lean · 2026-10-26
- Building AI bots on raw APIs? Devs say models are 'dumb without the harness' — madeofAjala · 2026-10-26
- HarnessMM cuts agent tokens per task by up to 88% while raising success rates — omarsar0 · 2026-10-26
- Microsoft demos intent-classifying agents built with Copilot App and Foundry, deployable across Teams and Outlook — marlene_zw · 2026-10-26
- Looking for a wearable mic with best battery life that feeds recordings to personal agents via MCP — altryne · 2026-10-26