Hypertune open-sources KV-cache soft-prefill adapters to serve heterogeneous fine-tunes in one batch

JoshPurtell · x · 2026-11-18

Hyperstition open-sourced Hypertune (MIT), claiming that serving individually fine-tuned models at scale is now economically feasible. The adapter is a "soft prefill" — trained key/value entries living entirely in the KV cache of a frozen model — so a single batch can mix many fine-tunes with default-model requests without LoRA's per-matrix bandwidth overhead, solving co-residence without capability regression. Integration means prepending the soft prefill to paged KV blocks in an existing serving stack. Josh Purtell argues that with autoresearch agents reaching SOTA on arbitrary setups, zero-penalty adapted inference should be standard.

Related event: Developers Explore New Paths Beyond LORA for Cheaper LLM Inference(2 posts)→

Original post →

More from coding & agent

coding & agent channel →