Open-Sourcing fastapi-stream-lease: Redis Leases to Stop Duplicate LLM Streams
Calm_Exchange_989 · reddit · 2026-10-28
The author open-sourced fastapi-stream-lease (MIT, no paid tier) to fix a common production leak when serving LLM streams over SSE/WebSockets: a second tab, reload or auto-reconnect leaves the first generation running on another worker, so one user burns tokens/GPU on parallel streams. Per-minute rate limiters can't catch this — the real question is how many active streams a user has open across all workers.
How it works: Redis sorted sets + atomic Lua scripts enforce active-stream concurrency limits across FastAPI/Starlette workers.
- Acquire a short-TTL lease before the generator starts (scoped per user/IP/account).
- Over the limit? Return 429 with Retry-After before any LLM call.
- A background heartbeat renews the lease mid-stream; release on completion/disconnect; TTL expiry covers worker crashes (OOM/SIGKILL).
- Supports WebSocket sessions, graceful drain on deploy, and configurable fail-open/fail-closed if Redis goes down.
v0.6.4, CI tested against Redis 5/7/8 plus Sentinel and Cluster failover, 100% line/branch coverage. Local demo: docker compose up --build demo, open three tabs at localhost:8080.
More from coding & agent
- Sitzmann open-sources texwerk, a self-hosted collaborative LaTeX editor with native agent support — vincesitzmann · 2026-10-29
- Professor runs DeckWerk's headless server mode for real-time lab slide collaboration — vincesitzmann · 2026-10-29
- Rime launches Prism, a speech-to-speech model that kills dead air in voice agent calls — testingcatalog · 2026-10-29
- "Terminal debt": devs avoid rebooting because resuming agent sessions and servers is a pain — mobileraj · 2026-10-29
- Solo Founders Podcast ep 34: agents now write, review and test the code — julianweisser · 2026-10-29
- Scoping Is Now the Bottleneck in ML Projects, Says Louis Dorard — louisdorard · 2026-10-29