NInfer v0.3.0 runs Qwen3-27B on a modded 22GB 2080 Ti, now Windows-native at 0.28GB
xrailgun · reddit · 2026-10-30
NInfer's Turing fork (v0.3.0) targets dense Qwen3.6/8-27B on a single modded 22GB RTX 2080 Ti, VRAM-only with no system RAM offloading. New in v0.3.0: native Windows support (no more Linux/WSL2/Docker), prebuilt releases, and aggressive code-path pruning (5090/FP8/FP4/35B-A3B removed) that shrank binaries from 1.56GB to 0.28GB and build times from hours to minutes, with bit-identical outputs.
Features and benchmarks (Win11, 64k context INT8 KV cache, weights 15.4GiB):
- OpenAI- and Anthropic-compatible APIs plus CLI; image/video input, thinking mode, CUDA Graph decode, MTP speculative decoding on by default
- Prefill: 616 tok/s on a 512-token prompt, 458 tok/s at 64k
- Decode: 27.0 tok/s baseline; with MTP, 47.5 tok/s on real code prompts (vs 26.3 without) but only 28.8 on prose — MTP helps code a lot, prose barely
Apache-2.0, derived from Neroued/ninfer.
More from Infra
- 2 GPUs pull 1,600W, 4 GPUs ~3,000W: check your circuit before buying — dee_hw · 2026-10-30
- Open-source Edge0 claims 35B model runs offline on iPhone in 1-2.5GB of memory — FinanceYF5 · 2026-10-30
- Hyperscaler capex crackdown: 11 of 16 AI stocks down 19%+, GOOGL RSI 15.6 — pdamodaran · 2026-10-30
- AMD's chiplet architecture winning as Intel misjudged memory-price hit to PC demand — zephyr_z9 · 2026-10-30
- Kentucky data center hearing descends into chaos as residents blast its noise — TatersAndMac · 2026-10-30
- NY data center moratorium was summer's most-lobbied bill as spending hit $64.5M — news-10 · 2026-10-30