2026-07-27
DKV is a KV-cache compression runtime that reduces each 256-token block to an anchor token, a rank-32 low-rank delta, and a budget of exact residuals, pushing Qwen2.5-1.5B to 64k context on an 8.6 GB Apple M3 (where default PyTorch OOMs at 16k) at the cost of slower decode.
The bottleneck in long-context inference is often not the model weights but the KV cache (the key-value pairs Transformer inference stores per token): it grows linearly with sequence length and is the first thing to exhaust memory on commodity hardware. A 1.5B int4 model weighs about 1 GB, but its KV cache can easily exceed that. The default full-KV PyTorch configuration on an 8.6 GB Apple M3 runs out of memory at just 16k context.
DKV targets exactly this. Without touching weights or precision, it turns the KV cache from a linearly growing pit into a bounded fixed pool, so long context runs on small local memory. The author is Om Chimurkar of Newton School of Technology; this is a technical report.
The core idea is to separate the smooth part from the outliers. Each block of 256 tokens is compressed into three pieces.
At decode time a fused routed kernel keeps the work bounded. A cheap router selects the top-K relevant blocks, scores the query in the low-rank subspace without decompressing K, attends the residuals and a recency window (the newest 1,024 tokens, kept exact) directly, and merges the two halves with a flash-style log-sum-exp reduction. Prefill uses training-free block-sparse attention, so its cost grows sub-quadratically.
All numbers are measured on one 8.6 GB Apple M3 with Qwen2.5-1.5B (int4).
| Metric at 64k context | DKV | Optimized dense | Default PyTorch dense |
| KV cache footprint | 1.12 GB | 1.88 GB | OOM |
| Prefill time | 477 s | 821 s (1.72x slower) | OOM at 16k |
| Decode throughput | 21.4 tok/s | 20.2 tok/s | OOM |
| Buried needle recalled exactly | Yes | Yes | / |
Headline figures: per-block compression is 1.44x at R=128 and 2.25x at R=64; DKV recalls a buried passcode exactly at every context from 4k to 64k; a cross-architecture recall check runs on Llama-3.2-3B. At 64k, prefill is 1.72x faster than the weight-matched optimized dense baseline.
KV-cache compression is the deciding factor for whether local long-context inference is viable at all. DKV's positioning is clear: it is a memory-and-prefill instrument, not a free lunch. Where the dense cache still fits (through 32k on this host), dense decodes faster and is the right choice. DKV's home turf is the regime where the cache would otherwise not fit, larger models, longer contexts, tighter memory, more concurrent sessions, and there the alternative to slower decode is no decode at all.
The author states the cost in full and hides nothing. That kind of balanced reporting is rare in KV-compression work.
Honesty is this report's signature, and the limits are the author's own.
Decode speed is the headline cost. The sparse representation must be reconstructed and merged every step, more per-token arithmetic than one fused attention; the E6 ablation pins the gap here. The most effective fix is a single fused decode kernel, not a change to the representation.
The CUDA/Triton path is written but not benchmarked. The repo contains Triton and C++/GGML decode kernels, but the evaluation host has no NVIDIA GPU, so there are zero GPU numbers and GPU performance is a hypothesis. Two correctness bugs (a metadata desync and a Triton residual-alignment issue) are fixed on CPU; GPU certification is pending.
Evaluation is narrow. Every performance number comes from one model (Qwen2.5-1.5B int4) on one host; correctness covers only needle recall and zero perplexity drift, with no RULER or LongBench and no head-to-head against quantization, eviction, or other low-rank compression. The residual budget and router are tuned for Qwen2.5-1.5B and unvalidated elsewhere. The fixed 256-block pool caps compressed capacity at 65,536 tokens.