Schema retrieval judged by recall alone misleads agent builders: author proposes per-token coverage and cost metrics
burkov · x · 2026-10-14
As AI agents access business data through repeated reasoning-and-query cycles, the schema information sent to the LLM becomes a major cost and performance bottleneck: large catalogs burn tokens, add latency, and distract the model with irrelevant tables and columns.
The author argues that schema-retrieval systems are commonly judged by recall alone, which makes sending the entire catalog look ideal despite being expensive and less effective in practice.
The article proposes three complementary evaluation measures:
- Useful schema coverage per token
- Total token use across a multi-step task
- Estimated cost per reasoning step
Takeaway: the metric mismatch can lead organizations to pick systems that cost more and work worse.
More from coding & agent
- ServiceNow and McGill's AgentHorizon benchmark shows LLM judges miss subtle agent failures — rohanpaul_ai · 2026-10-14
- Sazabi launches 1.0: an AI agent that reads your logs and alerts with root cause and fix — rohanpaul_ai · 2026-10-14
- OpenCrewBot: open-source Grok Bot alternative runs your own team of AI bots — Arindam_1729 · 2026-10-14
- Jev via Jinja-templated Skills cuts Claude context cost 70% and runs 4-5x faster — oleg_ivye · 2026-10-14
- Dev behind $1.27M AI DM operation details prompt drift and silent API degradation — alanshore222 · 2026-10-14
- Open-source MCP verifies Australian ABNs against the live Business Register — Impossible_Bit_2676 · 2026-10-14