Spomin: live KV cache compaction squeezes 500k tokens of context into 180k resident

wgaca2 · reddit · 2026-09-11

Developer alekk89 released Spomin, an experimental router that sits between harness and runtime and replaces history chunks with summaries directly inside the KV cache, letting long-running agent sessions continue without stopping for full compaction and reprocessing.

How it works

Original transcript preserved in chunks; cache space managed for system prompt, recent context, summaries, generation and recall.
A separate worker summarizes chunks in parallel (separate hardware or between main-model generations).
Eligible chunks are swapped for summaries with RoPE positions adjusted; the retained suffix is never re-prefilled — only inserted summaries are processed. Supports pause/resume of native generation, on-demand recall of original text by context ID.

Results

10:1 compression: 400k source tokens fit as 40k summary; with a 20k system prompt and 120k recent context, 500k+ tokens of source material within 180k resident tokens.
Tetris: near-equal quality (16/16+11/12 vs 16/16+12/12) in 69.8 vs 149.6 minutes, 22.6% fewer tokens. Research: 37 vs 57/100 managed, but 75.2% less primary input and 60.7% less active time.

Limits

Tested only with Qwen 3.8 27b; Qwen3-family architectures should work, Qwen 4 and MTP unsupported. Worker is a fine-tuned Qwen2.5 3B proof of concept (also tested Qwen3.5 9B). Experimental — keep defaults unless experimenting.

Original post →

Spomin: live KV cache compaction squeezes 500k tokens of context into 180k resident

More from Infra