Together AI and Ollama explain how prefix caching keeps long agent sessions fast
togethercompute · x · 2026-11-17
Zain Hasan (Together AI) and Parth Sareen (Ollama) co-authored a deep dive on making open models work for long-horizon agentic workloads.
The traffic pattern
- Every agent step re-sends the entire conversation: system prompt, all tool definitions, the request, every file read, every command run
- The return is usually a few dozen tokens of tool call — large ISL, small OSL
- Without prefix caching, long sessions get slower and costlier each turn
What each side must get right
- Harness: keep the prompt prefix untouched so cached prefixes stay reusable
- Inference engine: route each turn back to the GPU already holding that prefix cache
The piece also introduces Together's Polymorphic Inference, Post-Training, and Link offerings.
More from coding & agent
- I ran 3,360 experiments to see if AI agents recommend the right products — santiviquez · 2026-11-17
- Recreating The Godfather's office scene with MiniMax H3 in ComfyUI: 3 shots, 37s — hucun16 · 2026-11-17
- Seroter's reading list: humanizer skills tested, subagents considered harmful, token waste — rseroter · 2026-11-17
- Engineer's 2026 job: feeding useful tweets to your agent; Artifacts repos now test locally — itsOmSarraf_ · 2026-11-17
- OpenAI to host NYC hardware hackathon: build working prototypes with Codex in one afternoon — OpenAIDevs · 2026-11-17
- Jeremy Howard reimplements MIT's Recursive Language Models paper inside SolveIt, more concise than original — jeremyphoward · 2026-11-17