Stop button often doesn't stop the LLM: tokens keep generating unless the app hangs up
No-Peanut-6988 · reddit · 2026-10-20
Sorami measured what happens when a user presses stop in an AI app: the model only stops immediately if the app actually hangs up the connection. If it merely hides the output while staying connected, the LLM keeps writing in the background, burning GPU time nobody sees.
Findings from their lab setup:
- On hang-up, generation stopped immediately with nothing extra generated.
- With the connection kept open but unread, generation continued.
- On timeout-and-retry without hanging up the first attempt, both requests ran concurrently; on-time answers nearly halved.
The fix is simple: make sure your stop button and retry logic close the connection, not just hide output. Scope: self-hosted engines (vLLM, SGLang) plus two AI gateways; OpenAI's hosted service was not tested.
More from coding & agent
- Meshy plugin for Hermes Agent adds full MCP & Skill support with in-chat 3D model preview — Teknium · 2026-10-20
- NVIDIA releases 7,878-example agentic finance SFT dataset for tool-calling agents — kalyan_kpl · 2026-10-20
- Does a smaller local model make a useful prompt injection detector? — Plastic_Secret1303 · 2026-10-20
- Blogger says Jev alone can't spot AI replies, human feedback still needed — vista8 · 2026-10-20
- Princeton paper: combining specialized agents from many developers hits 70% on GAIA — rohanpaul_ai · 2026-10-20
- Agent eval blind spot: revoke access mid-conversation and test what leaks — jylusdev · 2026-10-20