Armin Ronacher explains how reasoning effort lives in the system prompt and wrecks KV cache
mitsuhiko · x · 2026-08-20
Armin Ronacher (creator of Flask) published "What Is Reasoning", prompted by a paper showing reasoning traces can be extracted from closed-weight models:
- Reasoning traces are just text: the model is trained to emit thinking into a scratchpad before its final answer. In GPT-OSS's Harmony format, the analysis and final channels use the same text separated by special tokens; a parser routes reasoning into a separate API stream, while closed models presumably use another model to redact and summarize it.
- Reasoning effort is baked into the system prompt: earlier APIs exposed it as a token budget, making it look like a sampling parameter, but it's really part of the system prompt.
- The post also explains why changing reasoning effort invalidates the KV cache and how CoT traces can leak—valuable reading for anyone optimizing inference serving.
More from Models
- llama.cpp dflash2: Qwen 3.8 27B Inference Speed Up to 3x — Top-Eye-8104 · 2026-08-20
- Dev reports: Qwen 3.8 27B only works with reasoning_level set to low — andrejusb · 2026-08-20
- Grok's free tier is real but unstable with changing caps — heypearlai · 2026-08-20
- Hack: Get free uncapped access to 400+ models via LMArena — heypearlai · 2026-08-20
- ChatGPT free goes unlimited; Claude free tier offers Sonnet 5 — heypearlai · 2026-08-20
- Qwen 3.8 27B hits 70 t/s with specific config parameters — dsdt · 2026-08-20