How to shrink a multi-thousand-token extraction prompt without losing accuracy or speed
Slow-Business8503 · reddit · 2026-09-16
A developer building an API-based structured extraction system is asking how to shrink a system prompt that has ballooned to thousands of tokens of JSON schema, field rules, conditional logic, defaults, and normalization rules — without sacrificing accuracy, latency, or token limits.
They list 10 candidate approaches:
- Static instructions in the system prompt plus provider prompt caching
- Leaner instructions paired with structured JSON schemas / structured outputs
- Splitting rules into task-specific prompts
- Few-shot examples instead of enumerating every edge case
- Preprocessing/normalizing input before the LLM call
- A small model for classification/routing with a larger model for extraction
- Storing rules externally and retrieving only relevant ones (RAG-style)
- Fine-tuning so repetitive instructions no longer need to be sent
The core trade-off is between token cost, latency, and preserving edge-case logic — a directly useful discussion for engineers building LLM extraction pipelines.
More from coding & agent
- Transformers.js v4.3.0 ships structured output: force exact JSON Schema in the browser — nicodotdev · 2026-09-16
- Agents slow down as memory notes pile up, unlike humans who speed up with practice — sujingshen · 2026-09-16
- 5 Open-Source AI Agent Repos Exploding on GitHub Right Now — Shruti_0810 · 2026-09-16
- Dev puts Codex voice in CarPlay via iOS app, codes during road trips — athyuttamre · 2026-09-16
- Developer builds an LLM decision API: parallel questions, probabilities and scores for code logic — DevDminGod · 2026-09-16
- Neko Master: open-source dashboard for real-time local gateway traffic auditing hits 4k stars — tom_doerr · 2026-09-16