Custom inference engine pushes Qwen3.8-Flash-Next to 65 tok/s on 12GB RTX 5070
KnownAd4832 · reddit · 2026-09-25
A Redditor built a custom inference engine (Strata, open source, one-click install, CUDA-optimized) after getting only 15 tok/s with llama.cpp on a 12GB RTX 5070. The same IQ3XXS quant of Qwen3.8-Flash-Next now runs at 65 tok/s output and 430 tok/s prompt processing. On 128K context with 64GB DDR5 + Ryzen 5 7600: Q20 at 65.1 tok/s out / 543 pp, IQ2XS at 52.0 / 472, IQ3XXS at 44.8 / 414. Minimum RAM+VRAM: 37.6GB (Q20) to 47GB (IQ3XXS), plus 0.91GB for the vision encoder. Model: ISTA-DASLab GGUF quants.
More from Infra
- Anthropic paid $5B for short-term compute in Q3 as spot prices run ~2x long-term deals — tengyanAI · 2026-11-11
- UK Sovereign AI invests in Fractile to build next-gen frontier AI chips — charlieharris01 · 2026-11-11
- What happened when I moved vector retrieval into Snowflake SQL instead of orchestrating RAG outside — dima806_dima · 2026-11-11
- AI chip startup Fractile raises $720M Series C for DRAM-on-processor design — Tom_Westgarth15 · 2026-11-11
- Mobile AI Factory: agentic pipeline builds on-device SLMs that beat 5-shot frontier models — kalyan_kpl · 2026-11-11
- Vivo's Starlink direct-to-cell service hits 10M users in two weeks across Brazil — XFreeze · 2026-11-11