NVIDIA Paper Caches LLM Reasoning as CPU Code, 16-291x Cheaper
rohanpaul_ai · x · 2026-11-22
A new NVIDIA paper introduces "agentic reasoning caching": once an LLM has reasoned through a problem, the solution can be compiled into CPU-executable code and reused for rephrased or re-numbered variants, cutting invocation cost 16-291x versus the source model.
- Method: Claude Opus 4.5 converts a reasoning trace into small Python functions with changeable inputs as parameters; at runtime these run on CPU, with a fine-tuned 1B model called only for small text jobs like value extraction.
- Results: the cache matched or beat the source model on 7 of 8 math and science benchmarks, scoring 93.3% vs 83.3% on AIME '25, and handled at least 90% of rephrased or re-numbered variants.
- Takeaway: agents that run the same prompt on new inputs all day should cache reasoning as code and call the LLM only on misses.
Related event: NVIDIA's CPU-ARC Caches LLM Reasoning as Code, Cutting Costs Up to 291x(2 posts)→
More from Infra
- SK Hynix reportedly tops 70% yield on HBM4, starting mass production in Q3 — Beth_Kindig · 2026-11-22
- Mflux Runs MOSS Image Gen on 8GB Macs: 82s for 768x768 With 4-Step Distill LoRA — bio_risk · 2026-11-22
- After a Possible Riemann Proof, a Call to Reinvent AI Memory and Inference Cost — Kitchen_Mirror_7247 · 2026-11-22
- Intel D270 AI card: 160GB LPDDR5X, 350W, ~$10K, and why call it a GPU? — Terminator857 · 2026-11-22
- Speed-to-power dominates: on-site solar-plus-storage cheapest for AI data centers in 85% of US locations — mattbeane · 2026-11-22
- LexiEngine, an AMD-focused Rust inference engine, adds Inkling support — W61k3r · 2026-11-22