Armin Ronacher explains how reasoning effort lives in the system prompt and wrecks KV cache
mitsuhiko · x · 2026-08-20
Armin Ronacher (creator of Flask) published "What Is Reasoning", prompted by a paper showing reasoning traces can be extracted from closed-weight models:
- Reasoning traces are just text: the model is trained to emit thinking into a scratchpad before its final answer. In GPT-OSS's Harmony format, the analysis and final channels use the same text separated by special tokens; a parser routes reasoning into a separate API stream, while closed models presumably use another model to redact and summarize it.
- Reasoning effort is baked into the system prompt: earlier APIs exposed it as a token budget, making it look like a sampling parameter, but it's really part of the system prompt.
- The post also explains why changing reasoning effort invalidates the KV cache and how CoT traces can leak—valuable reading for anyone optimizing inference serving.
More from Models
- Grok Bot Ran a 12-Hour Data Analysis Nonstop, Handling API Rate Limits Itself — Daniel_Farinax · 2026-10-16
- One year of Recursive Language Models: GPT-5-mini beats GPT-5 by 110% on 132k-token tasks, cheaper too — a1zhang · 2026-10-16
- Users Report Claude Models Suddenly More Compliant After Forced Compaction — RealSharpNinja · 2026-10-16
- Prediction: Opus 5.5-level models on laptops by 2027, backed by Qwen 27B data — cephaloform · 2026-10-16
- Users Say Krea 2 Output Degraded Sharply Since Krea 3 Beta Launched — szywis · 2026-10-16
- LLM-as-a-Verifier lifts Nemotron 3.5 Super by 9-10 points on agentic benchmarks — simonguozirui · 2026-10-16