Why Is Q&A on Large Documents Still So Fast
Economy-Builder7916 · reddit · 2026-07-13
This in-depth technical post explains why dropping a 40-page PDF into ChatGPT still yields seemingly instant response times.
The author outlines common inference acceleration factors:
- Streaming output: The model generates and returns text simultaneously, lowering perceived latency.
- KV caching: Previously processed context isn't recalculated.
- Quantization: Lowering compute precision in exchange for faster inference.
- Speculative decoding: A smaller model guesses the next tokens, verified by the larger model.
- Batching & hardware optimization: Running multiple requests together to boost GPU utilization.
Regarding why large documents don't significantly drag down speed, the author points to:
- Input phase parallelism: The prefill phase can process the entire context at once, unlike the token-by-token decoding phase.
- Chunking/retrieval mechanisms: The system might not brute-force the entire document into the context window every time.
- Cross-turn caching: Follow-up questions on the same document usually reuse previously processed results.
The post concludes with open questions: how much of ChatGPT's actual speedup comes from the architecture itself, product-level RAG/chunking, or raw compute power—and whether speculative decoding is now widely used in production.
More from Infra
- SkyPilot emerges from stealth with over $20M to tackle fragmented AI compute — skypilot_org · 2026-07-22
- Why a 1GW Chinese AI data center may be plausible after all — teortaxesTex · 2026-07-22
- China’s AI arms race is increasingly defined by chips, data centers, and open models — BenBajarin · 2026-07-22
- Agent search bottlenecks are now about variance, not raw latency — rohanpaul_ai · 2026-07-22
- Gavin Baker argues Nvidia may be one of open source AI’s biggest supporters — GavinSBaker · 2026-07-22
- AI Power Demand Exposes US Energy Gap, Urging Shift from Scarcity to Abundance — bradneuberg · 2026-07-22