OpenJet runs 4 coding agents on one RTX 3090 at ~290 tok/s via custom CUDA megakernel
Adorable_Weakness_39 · reddit · 2026-10-27
OpenJet's new release serves multiple coding agents concurrently on a single RTX 3090 with a custom CUDA megakernel for Qwen3.8-27B Q4KM, totaling 256-288 tok/s vs 72-80 for llama-server --parallel 4.
- Per-agent decode: 64-72 tok/s short-context, 52-55 at 23K context; next-request first token drops from 13.4s to 0.2-0.5s
- Key insight: decode is memory-bandwidth-bound — the engine stacks all agents' tokens plus MTP/prompt-lookup drafts (12 rows for 4 agents) into one batch, so weights stream once per step instead of four times
- Engineering details: paged shared 132K q80 KV pool, shared system prompts stored once, chunked prompt processing that doesn't stall other agents (bit-identical results), sampled MTP drafts with rejection sampling, and an mktop live monitor
- Requires Linux, 24GB RTX 30/40/50, CUDA 12+; exposes an OpenAI-compatible endpoint
Related event: Single RTX 3090 runs 4 coding agents at ~290 tok/s with custom CUDA engine(2 posts)→
More from coding & agent
- Nous Research Revamps Hermes Agent Plans: Cloud Agent Included, Daily/Weekly Limits Replace Monthly Credits — NousResearch · 2026-10-28
- Harmonic Launches Aristotle for Automated Formal Verification of C and Verilog Code — satnam6502 · 2026-10-28
- Verifiers have bugs too: flaws found in QED prover and RISC-V Lean model — xennygrimmato_ · 2026-10-28
- Anthropic's official Claude skill keeps your CLAUDE.md up to date automatically — udmrzn · 2026-10-28
- 5 open-source repos for giving AI eyes and hands on your computer — udmrzn · 2026-10-28
- Infini-AI-Lab open-sources Hermit: continual learning via reusable adapters, no model changes — udmrzn · 2026-10-28