This WhatsApp bot feeds 252K tokens to the LLM on every 'hi' — and rejected the 4.6x cheaper retrieval version
louay_Sallakho · reddit · 2026-09-16
An engineer running a WhatsApp support bot for 27,000 customers explains why every message — even a bare "hi" — gets the full 252K-token policy set with zero retrieval. The n8n-assembled prompt has four parts: all policies, 320 pre-filled live system values, 130 conditional policy blocks resolved before the model is called, and separate LLM guardrails reviewing replies and tool calls.
They also built a retrieval version of the identical bot: a classifier picks relevant sections, averaging 55K tokens — 4.6x cheaper. They kept the expensive one anyway, arguing retrieval fails silently: pull the wrong section and the bot still sounds fully confident, and you learn of it weeks later from an angry customer. Injecting everything removes the lookup step that can go wrong.
Open questions they raise: how badly does instruction-following degrade at 250K depth, does a rule at 80% depth still get obeyed, and how do you keep such a giant prompt maintainable (versioning, review, regression testing) — which they find harder than the token cost.
More from coding & agent
- Graphify, a 118K-star repo, turns codebases into queryable knowledge graphs for coding agents — techNmak · 2026-09-16
- Postgres pro tip: set application_name for human-readable query sources — DanielLockyer · 2026-09-16
- AI training at an energy company shows domain expertise doesn't teach agent engineering — NumbersProtocol · 2026-09-16
- Ministral 3 3B on a Galaxy S21 relays chats between Gemini and Z.ai across two browsers, 10/10 runs — Mean-Standard7390 · 2026-09-16
- Synapse: an open-source TypeScript gateway unifying multiple MCP servers behind one login — Comfortable_Cat_6207 · 2026-09-16
- Meta ships WhatsApp Business Tools MCP server beta for AI agents — shauntrennery · 2026-09-16