TokTier speeds up agent serving with exact stateful tokenization and stable-boundary repair
omarsar0 · x · 2026-08-04
TokTier is a stateful tokenization service built for agentic LLM serving, designed to preserve exact tokenization while making session continuations far faster.
- In real agent traffic, tokenization can consume up to 64% of time to first token because coding agents repeatedly resend long transcripts after every tool result.
- TokTier re-tokenizes only a small window around the append, checks for a stable boundary, and splices only when that boundary is safe.
- If no reusable prefix exists, it falls back to exact pre-tokenization and GPU-based BPE.
- The paper reports zero divergence across large differential test campaigns, incremental repair latency of 0.5–1.1 ms, and up to 437× speedup over Hugging Face in some ranges.
More from coding & agent
- Eve pushes enterprise agents toward one customizable assistant per company — aarthir · 2026-08-04
- Enkstein open-sources a local control plane for Codex, Claude, Gemini, and Ollama — wcoreiron · 2026-08-04
- Gemini Spark can connect MCP servers, but every tool still needs repeated approval — tristanbob · 2026-08-04
- Agent apps should expose a skill surface, then crystallize it into code — burny_tech · 2026-08-04
- Use subagents to switch models inside one session — dotey · 2026-08-04
- xAI updates Grok Build with Grok 4.5, skills, MCP, and plan mode — elonmusk · 2026-08-04