Ignis inference server extends Qwen3.8-27B local coding-agent serving to RTX 30/40 cards
gpillon · reddit · 2026-11-16
Developer gpillon released Ignis 0.10.0, extending his local inference server for Qwen3.8-27B from Blackwell to RTX 3090 and RTX 40 series cards, designed to sit on your dev machine sharing the GPU with your desktop rather than chase peak tok/s.
- Engineering choices: sizes context to free VRAM at startup and refuses to start if it won't fit; unloads the model after 5 minutes idle; keeps the vision tower in system RAM so vision costs no context when idle; runs the main agent and subagents concurrently on one card; resumes repeated prompts (system prompt, repo context) from cache; OpenAI-compatible (chat completions + Responses), single binary with browser Playground and monitor.
- Quantization: 30/40-series builds use 3-bit trellis quantization (13.4 GB download) with DFlash2 speculative decoding and vision; on 24 GB cards it allows 8 parallel requests and up to 1,048,576 tokens of context, 2 and 262,144 on 16 GB.
- Numbers: on a 5090 NVFP4 build, 370 tok/s single stream, 1,000+ tok/s at 8 concurrent requests, 11,000 tok/s prefill on a 46K prompt; a stranger independently ran it through airbench with opencode on a 5090, passing all 49 tests (math, vision, email, web shopping, coding) in 26 minutes.
- Caveat: the author doesn't own a 30/40-series card — the release gate passed on Ada (sm89) with bit-identical results to an earlier Ampere (sm86) build, 93.3% top-1 agreement with BF16, all 127 kernel tests passing — but real-world speed and whether 1M context fits remain unverified, so he's asking for testers.
More from coding & agent
- Ex-games founder prototypes a side-project game with AI: new arena and two characters — trq212 · 2026-11-16
- Qwen4-Max debuts #6 in Code Arena WebDev, makes Alibaba the top Chinese lab at #5 overall — arena · 2026-11-16
- Hologram lets Codex and Claude Code make real phone calls from your own number via MCP — Melodic_Builder_8170 · 2026-11-16
- LMArena launches Agent Arena leaderboard for comparing AI agents — arena · 2026-11-16
- New Maggie talk: design engineers are best positioned to prototype the future with LLMs — manosaie · 2026-11-16
- First-time dev ships an Android vehicle battle-royale with 5 maps and 7 characters using AI coding — AstraNoms · 2026-11-16