1M Token Context on Single RTX 3090 Achieved via KVarN Quantization
Anbeeld · reddit · 2026-08-10
Developer Anbeeld shared an extreme local LLM deployment test: a user successfully loaded nearly 1 million tokens of context for a 17GB Qwen 3.5 35B A3B model on a single RTX 3090 (24GB VRAM).
The test wasn't just about the server staying alive; the user successfully extracted 7 "needles" (precise information) placed in various parts of the massive text, proving effective long-context understanding under extreme quantization.
This breakthrough utilizes the BeeLlama.cpp fork and Huawei's KVarN (Variance-Normalized KV-Cache Quantization). Compared to standard 4-bit quants, KVarN demonstrates superior precision at ultra-low bitrates, fundamentally changing expectations for low-bit KV cache quantization performance.
Related event: Meta Open-Sources 30B Muse Glimmer for On-Device Agents(108 posts)→
More from Infra
- Amazon Proposes Per-Request Validated Evidence Caching for Production RAG — _reachsumit · 2026-10-16
- LinkedIn's TransX report: sequence transduction for recsys, +6.02% CTR in A/B tests — _reachsumit · 2026-10-16
- llama.cpp PR boosts CPU inference speed with merged AVX2 tiled accumulators — jacek2023 · 2026-10-16
- Zero's next-gen IVM engine runs in a single DO, live-querying a 100GiB R2 database — threepointone · 2026-10-16
- Meme: driving to the chip factory in 2028 captures AI's compute anxiety — beffjezos · 2026-10-16
- One volunteer maintains the time zone file behind 4 billion phones — danbri · 2026-10-16