8 Coding Agents × 250K Context Each, Concurrently, on Two RTX 3090s (~600+ tok/s)
o2beast · reddit · 2026-11-02
Instead of TokenGate-style upfront token pruning, the author keeps the full int4 KV cache in system RAM and per decode step runs attention on only 1% of tokens, selected via a rank-64 key sketch. With custom Triton kernels, vLLM patches, and a RAM tier with prefix caching, two RTX 3090s serve 8 concurrent coding agents each holding 250K tokens at 545–790 tok/s total, 31/32 recall, and +15–41% speedup even for single long chats. Released as patches on top of HyperQwen with all bench scripts; caveat: approximate attention occasionally confuses near-duplicate code lines.
More from Infra
- Five-year compute map: NVIDIA holds 93% of GPU units, custom ASICs to pass GPUs by 2027 — pdamodaran · 2026-11-02
- Space datacenter heat is just engineering; cosmic rays are the real challenge — shashib · 2026-11-02
- Louisiana's $300B+ project pipeline includes Meta Hyperion, Amazon and Hut 8 AI data centers — MatthewChang · 2026-11-02
- AI data centers destroy 250,000 trees, undercutting 'paperless' eco promises — im_mansigupta · 2026-11-02
- "Someone must solve the cost of serving coworking agents" — hyhieu226 · 2026-11-02
- SmolChat v17: On-Device GGUF LLMs on Android Now Ships an OpenAI-Compatible Server — shubham0204_dev · 2026-11-02