Training-free memory augmentation makes CoT cheaper: +21-29 pts and 1.5x speedup
dreamwieber · x · 2026-09-01
An arXiv paper, Memory Augmentation Unlocks Efficient Chain-of-Thought Reasoning, offers a new way to cut CoT costs: move reusable reasoning from generation into the prompt.
Key ideas:
- Instead of regenerating every reasoning step, the framework distills historical traces into reusable reasoning memories (summarizing patterns, key constraints, and critical operations), retrieves the relevant ones, and provides them as prefill-side scaffolds so the model thinks shorter.
- Training-free; the paper formalizes the trade-off as the "Context-Generation Substitution Law".
Results: Memory improves prompt-based Chain-of-Draft compression by 21.4 / 28.0 / 29.5 / 6.61 points on GSM8K, MATH, BBH, and MMLU-Sci, while achieving 1.14-1.49x latency speedup over standard CoT. It's compatible with token-level, trace-level, and inference-state compression.
Related event: Training-Free Memory Augmentation Makes Chain-of-Thought Cheaper and Faster(2 posts)→
More from coding & agent
- Code should be explainable, not necessarily readable, in the post-AI era — kieranklaassen · 2026-09-01
- Dev uses AI-generated textbooks with custom tool for interactive learning — generativist · 2026-09-01
- Agents Cannot Read LinkedIn: Social Platforms Block AI Crawlers via Robots.txt — Dry_Steak30 · 2026-09-01
- State-Driven Protocol Stops LLM Over-Fixing and Context Collapse Without Vector DB — wenger2026-12 · 2026-09-01
- Using Rust Runtime Constraints to Guide AI Agents in Writing Concurrency-Safe Code — doodlestein · 2026-09-01
- Run Qwen 27B on 16GB VRAM: llama.cpp MTP mod adds 17% speed, more context — ea_man · 2026-09-01