Chatbot injects 252k-token prompt every turn instead of RAG, dev asks to be talked out of it
louay_Sallakho · reddit · 2026-09-15
A developer running a WhatsApp support bot (27,000 customers) shares a deliberately extreme architecture: every message gets a 252,000-token prompt with no retrieval at all.
Prompt structure
- Main prompt with all policies
- 320 live ERP parameters (balances, contract status, dates) injected before the model sees anything, so it never needs to look anything up
- 130 conditional policy blocks: if/then branches evaluated by Python at prompt-build time, not by the model — only the matching branch gets injected
- Separate LLM guardrails reviewing replies before customers see them
Why not RAG: they also run a retrieval version (55k tokens, 4.6x cheaper) but distrust it because retrieval fails silently — a classifier grabbing the wrong policy section yields confident wrong answers nobody notices. Sending everything removes the failure point.
Open questions posed: attention degradation at 250k tokens (are rules buried 80% down actually followed?), whether 'deciding branches at prompt-build time' has a proper name, and how others handle prompt versioning and rollback. Stack: Notion + n8n.
More from coding & agent
- ElevenLabs Brings Voice, Music, Image and Video Generation to Its MCP Server — nikola_mr64990 · 2026-09-15
- Consultant shares an 'as needed' engineering practice plus a purpose-built skill — _rchaves_ · 2026-09-15
- Kimi Code ships Remote Control by default, native Linux clipboard and plugin marketplace — KimiDevs · 2026-09-15
- Developer Shares Hands-On Use of a Claude Code Persistent Memory Skill — doodlestein · 2026-09-15
- Hermes Newswire, a zero-token RSS news ticker plugin, merged into NousResearch's official catalog — Teknium · 2026-09-15
- vphone-cli lets AI agents control a fully virtualized iPhone on Apple Silicon, +633 stars in a day — JiliJeanlouis · 2026-09-15