KVA projectors boost local LLM prefill up to 1.85x with tunable perplexity tradeoff
Public_Umpire_1099 · reddit · 2026-09-24
- A Redditor implemented KVA projectors (inspired by DeepSeek V4.1 Flash and HySparse2/MiMo-V3) for Qwen3.8 Flash Next, running locally on 2x R9700 with 128GB DDR5.
- Tunable speed/quality tradeoff depending on start layer: 1.85x prefill speedup (1700→3150 t/s) at +8% perplexity from layer 12; 1.7x at +5% from layer 16; 1.45x at +2.6% from layer 24.
- Unlike other implementations: one full per-layer map using Tikhonov/ridge regression (1.5GB VRAM), predicting only later-layer inputs while real weights compute keys/values, with a selectable start layer instead of a fixed layer 24. The author found extra training nearly worthless — it's architectural levers that matter.
- A second multi-layer version achieves 1.55x speedup at only +2% perplexity, with a projected path to 1.5x under 1% ppl increase. V1 is stable and usable in R9V now; V2 is behind a --ced quality flag. Both the IQ4XS model and projector repos are on Hugging Face.
More from Infra
- Investor Predicts EDA/CAD Will Collapse Into One Flow Within 3-5 Years — ai · 2026-09-25
- AI Data Center Debt Starting to Roll Over, Rising Rates Accelerating the Problem — AIFlow_ML · 2026-09-25
- Musk details xAI compute: Colossus 2 to hit 880k GB300s by year-end — elonmusk · 2026-09-25
- Qwen-Image-2.1 gets GGUF quantization, could run text-to-image on a Snapdragon 865 phone — ResidentAping · 2026-09-25
- Deep Inference-Query Engine Integration: Custom Scheduler and Workload-Aware KV Cache for Prefill-Only AI Filters — charles_irl · 2026-09-25
- Finance worker seeks local AI setups to cut soaring Codex/ChatGPT costs — Startup__Sam · 2026-09-25