Inference for fine-tuned models is underexplored: LoRA vs multi-tenant KV debate

JoshPurtell · x · 2026-11-18

Josh Purtell lays out three possible paradigms for serving fine-tuned LLMs: properly solving LoRA inference (which he thinks is increasingly unlikely), developing new approaches like soft prompts or multi-tenant KV sharing (a repo with promising early results, though undecided), or using swarms to max out single-tenant GPU load.

He adds that ideally both full fine-tunes and multi-tenant KV should be high-performance, and argues that while tons of people work on kernels for un-adapted base models, there's a lot of room for serious work on inference for adapted models.

Original post →

More from Infra

Infra channel →