Hypertune open-sources KV-cache soft-prefill adapters to serve heterogeneous fine-tunes in one batch
JoshPurtell · x · 2026-11-18
Hyperstition open-sourced Hypertune (MIT), claiming that serving individually fine-tuned models at scale is now economically feasible. The adapter is a "soft prefill" — trained key/value entries living entirely in the KV cache of a frozen model — so a single batch can mix many fine-tunes with default-model requests without LoRA's per-matrix bandwidth overhead, solving co-residence without capability regression. Integration means prepending the soft prefill to paged KV blocks in an existing serving stack. Josh Purtell argues that with autoresearch agents reaching SOTA on arbitrary setups, zero-penalty adapted inference should be standard.
Related event: Developers Explore New Paths Beyond LORA for Cheaper LLM Inference(2 posts)→
More from coding & agent
- Microsoft Ignite LAB663: a hands-on loop for improving agents with Copilot traces and Model Router — lee_stott · 2026-11-18
- Agent tweak is 15% faster at nearly same cost — the model admitted it overweighted raw token counts — dosco · 2026-11-18
- Aware browser bets on feeds, not chat, as the right interface for AI agents — HaijunXia · 2026-11-18
- Why Cheaper Tokens Can Still Mean a Bigger AI Bill: The Agent Call-Stacking Problem — samcharrington · 2026-11-18
- DAWE edits 5,064 of 4B weights to fix harness mismatch, no retraining needed — lateinteraction · 2026-11-18
- Why the 15-minute time limit: bench author says it matches median agent runtime in prod — JoshPurtell · 2026-11-18