Running Hy4-Flash locally on 2x RTX 3090: Q3 quant hits PP 63 / TG 22 tokens per second
EmilPi · reddit · 2026-11-19
The author benchmarked AngelSlim's Hy4-Flash-GGUF Q3K quant on llama-server with 2x RTX 3090 + 128GB DDR4, calling it a genuine rival to DSv4-Vision-Exp and Qwen3.8-27B-Uncensored, finishing answers with fewer tokens despite being slower.
Key setup and results:
- Flags: --ctx-size 131072 -ctk q80 -ctv q80 --no-mmproj-offload -cmoe --moe-cache-mib 33768 (the last one is llama.cpp's expert streaming, inspired by Strata)
- Measured PP 60-65 tokens/s, TG 20-24 tokens/s
- Requires llama.cpp PR #32517, rebases cleanly on master
Full config file excerpt and command line included, easy to reproduce.
More from Infra
- OpenAI: median researcher spends $2,400/day on agents, token costs now 2x labor costs — daveholtz · 2026-11-19
- Fchollet: latent token demand is near-infinite, cost is the only brake as RoI keeps improving — fchollet · 2026-11-19
- Meta exec: without owning compute on US soil, economic and national security foundations crumble — BenBajarin · 2026-11-19
- OpenAI adds discounted background mode: save 30-40% per token on GPT-6.2 with 2h/8h windows — OpenAIDevs · 2026-11-19
- EchoMoE Sends Only Deltas Between Nodes, Cutting MoE Inference Traffic 58% — sivareddyg · 2026-11-19
- JEDEC unveils DDR6 spec: 12.8 GT/s起步、最高 14.4 GT/s,带宽提升 50%,支持 128Gb 颗粒 — gnukeith · 2026-11-19