Bolting on a small approximator halves Qwen3-8B prefill without retraining
rickasaurus · x · 2026-09-14
Borrowing DeepSeek V4.1 Flash's Causal Encoder-Decoder idea, researchers trained an external approximator module for Qwen3-8B that predicts the second-half KV cache, cutting prefill time nearly in half with identical outputs — no weight changes needed. If it generalizes, existing fine-tuned open models could get drastically faster local long-context deployment without waiting for next-gen releases.
More from Infra
- Nebius runs auction-driven GPU pricing, lifting rates ~20% amid tight supply to 2028 — BenBajarin · 2026-10-30
- macOS update silently re-adds eMudhra root certs users had removed — steipete · 2026-10-30
- Bittensor's Terza Inference routes one API across miners' models on subnet 103 — const_reborn · 2026-10-30
- SGLang Summit 2026 sells out sponsor booths and Full Passes ahead of Nov 12–13 event — ying11231 · 2026-10-30
- OpenRouter rebuilds provider routing: 46% fewer capacity errors, 16.9% faster TTFT — steipete · 2026-10-30
- NVIDIA's Nemotron 3.5 Super hits 51.78% on SWE-bench Pro with 4.6x fewer active params — dair_ai · 2026-10-30