One table shows speculative decode and agent loops are the same algorithm at different layers
akbirthko · x · 2026-10-21
A systems-thinking post argues the inference stack is isomorphic above and below the API:
- Serial work: draft speculation below vs. long CoT reasoning above
- Parallel work: target verification below vs. fast state decisions above
- Commitment: accepted tokens below vs. decisions enacted in reality above
- Revision: rejected continuations below vs. revised thoughts/plans above
- State: context/KV cache below vs. context, latents, world state above
- Loop: draft-verify-append below vs. observe-decide-act above
- Bottleneck: memory traffic and decode depth below vs. stale state and action deadlines above
- Economics: useful tokens per dollar below vs. useful actions per dollar above
The author extends this to async reasoners serializing search and state, "decision models" (effort level = 0?), and speculates about a future API for physical reality. A conceptual framing with an elegant mapping table, though no implementation details.
More from Infra
- Seroter's reading list: TorchTPU open source, Claude Code's code review system — rseroter · 2026-10-21
- NVIDIA quietly open-sources Flare, a Rust-based local inference server for Blackwell GPUs — AIFlow_ML · 2026-10-21
- Data center debt being repackaged to pension funds, likened to subprime bubble — SumitGup · 2026-10-21
- DeepInfer lacks KV caching, load balancing and disagg, but 3-4K GB200s can match DeepSeek's reported throughput — teortaxesTex · 2026-10-21
- Gemini 4 Argon tops Isoquant's inference benchmark with 4.38x mean GPU speedup — ycombinator · 2026-10-21
- Musk: SpaceX & Tesla's Terafab Aims to Mass-Produce Advanced AI Chips by 2028 — Polymarket · 2026-10-21