Kinference routing benchmark: cache-aware routing cut TTFT from 6.88s to 0.31s on shared prefixes
shoumikgoswami · reddit · 2026-11-08
Buoyant's Kinference sits in front of vLLM workers and routes requests using signals like cached prompt prefixes and queue size. In its own benchmark, median time to first token on shared-prefix traffic dropped from 6.88s with plain Kubernetes routing to 0.31s; without shared prefixes there was no meaningful difference (6.92s vs 6.83s).
Caveats:
- The comparison is against random pod routing, not another cache-aware router; both sides used three L40S GPUs running the same Qwen model
- These are vendor figures, not independent benchmarks
- Kinference is still alpha, with rate limiting and semantic classifiers listed as missing
The author's takeaway for self-hosted agent setups: before blaming the model for slow first tokens, check whether repeated system prompts and conversation history are landing on workers that can actually reuse the cache.
More from coding & agent
- Garbage in, garbage out: Opus makes bad videos from bad starting points — pramodk73 · 2026-11-08
- Dev shares his $1,000/mo AI coding setup after months of testing tools — jarrodwatts · 2026-11-08
- DeepSeek Harness 0.2.2 adds experimental sub-agent plugins for Codex and Claude CLI — sven_ai · 2026-11-08
- Thinking Machines serves Inkling 2 Small free on OpenCode for a limited time to gather feedback — clarejtbirch · 2026-11-08
- Agent swarms show endogenous division of scientific labor across 1,800 sessions — _AustinCalvert_ · 2026-11-08
- Dev builds self-hosting platform with Claude, benchmarks it against Dokploy — mynameiscorange · 2026-11-08