Agents that don't close connections on retry waste ~148 tokens per task, vLLM tests show
No-Peanut-6988 · reddit · 2026-10-23
Sorami measured what happens on self-hosted vLLM 0.31.0 when an agent client gives up on a request:
- Closing the connection: generation stops, 0 extra tokens.
- Stopping reads but keeping the socket open 5s: 44 extra tokens per cancelled request.
- Reading to the end and discarding: 53 extra, 175 on longer answers.
For retries (1.5s deadline + one retry): unclosed old attempts wasted 148 tokens per task vs 0 when closed; on-time completions fell from 38% to 21%, and to 11% if the abandoned connection stayed open, since it keeps competing for the same GPU.
Fix: actually abort the stream (AbortController, closing the response, cancelling the task) instead of just stopping the await. Tested only on vLLM/SGLang with L4/A10G and two models; llama.cpp, Ollama, TGI and hosted APIs untested.
More from coding & agent
- MIT Prof Built an AI Playbook That Audits Academic Websites and Simulates Visitors — soumitrashukla9 · 2026-10-23
- Robot arm Podrick learns to load utensils by distilling lessons from its own failed runs — ycombinator · 2026-10-23
- Agent sandbox test: auto-approve let exfiltration to new hosts through 12 of 12 times — No-Peanut-6988 · 2026-10-23
- For writing JSON schemas, Claude is the best pick and Gemini the worst, dev says — gaganghotra_ · 2026-10-23
- OSI Hackathon winner Consilith: an open multiplayer studio where researchers and their AI agents collaborate on science — BenBlaiszik · 2026-10-23
- Claims Engine: a graph-based system connecting evidence to claims — anselm · 2026-10-23