Local LLM rig: is dropping from Q6_K to IQ3_XXS Qwen 27B worth it for agentic work?
milpster · reddit · 2026-10-13
A Reddit user runs Qwen 27B Swift 1.5 at Q6K with 262K context and Q8/Q8 KV cache across a mixed 40GB VRAM rig (2x Radeon VII + RTX 3080 Max-Q via rocm/vulkan) plus 64GB DDR4, hitting 450tps prefill and 15-30tps output with dflash2.
They're weighing a move to the newer Flash variant at only IQ3XXS, unsure whether the heavier quantization would preserve "smartness" for agentic development and general agent tasks at comparable speed. Extra hurdle: the mixed vulkan/rocm backend rules out existing tooling, likely requiring a heavily modified llama.cpp fork. The post is an open question with no conclusions, but the rig details (quant tiers, KV compression, throughput numbers) are useful reference for heterogeneous multi-GPU local inference.
More from Infra
- Adobe ships NVIDIA RTX Spark optimizations: up to 2x faster creative apps and 5x Frame.io gains — shashib · 2026-10-16
- Meta's commitments jumped to $349B in a quarter as server memory prices tripled ahead of Oct 21 earnings — tengyanAI · 2026-10-16
- Meta releases Muse Spark Open, trained with vLLM-powered RL inference from day one — colinmcnamara · 2026-10-16
- a16z: bot 403 blocks double as Amazon cracks down on agentic commerce — a16z Newsletter · 2026-10-16
- Meta's open Muse Spark needs 493GB of memory — far beyond any consumer laptop — TejasKumar_ · 2026-10-16
- CoreWeave plunges 20% as credit markets price >60% default odds; is this dot-com all over again? — kimmonismus · 2026-10-16