Making Agents Fast: A Year of Lessons on Inference Speed and Caching
rseroter · x · 2026-11-03
Martin Alderson applies years of web performance optimization principles to agentic workflows, sharing key learnings from the last 12 months.
Inference speed splits into two familiar backend metrics:
- TTFT (time to first token): analogous to TTFB, includes prefill and scales with input tokens; lower is better. Prompt caching is vital here — cached tokens are faster and cheaper — though most agent harnesses already handle this well in practice.
- tok/s (tokens per second): how fast tokens stream once the response starts, akin to bandwidth; higher is better.
Relative importance depends on the workflow: agents exchanging many short turns are more sensitive to TTFT.
More from coding & agent
- LangChain's Interrupt agent conference returns to SF on March 24-25 with early-bird tickets — LangChain · 2026-11-03
- A daily agent prompt that turns your programs' status into slide decks and Gantt charts — nlarusstone · 2026-11-03
- Open-Source MCPlama Maps Your Entire MCP Permission Setup as an Interactive Network — kirto_am · 2026-11-03
- The rise of the design engineer: AI blurs the line between design and code — soleio · 2026-11-03
- DeepSeek releases all 7 demo prompts for Harness Creator Mode plugins — rohanpaul_ai · 2026-11-03
- Ex-ByteDance engineer on Coze shutdown: workflows killed by models that plan — dotey · 2026-11-03