Inference for fine-tuned models is underexplored: LoRA vs multi-tenant KV debate
JoshPurtell · x · 2026-11-18
Josh Purtell lays out three possible paradigms for serving fine-tuned LLMs: properly solving LoRA inference (which he thinks is increasingly unlikely), developing new approaches like soft prompts or multi-tenant KV sharing (a repo with promising early results, though undecided), or using swarms to max out single-tenant GPU load.
He adds that ideally both full fine-tunes and multi-tenant KV should be high-performance, and argues that while tons of people work on kernels for un-adapted base models, there's a lot of room for serious work on inference for adapted models.
More from Infra
- 594B model now runs on two DGX Sparks at 37.9 tok/s; drafter boosts 27B to 78 tok/s — AIFlow_ML · 2026-11-18
- Why Cheaper Tokens Can Still Mean a Bigger AI Bill: The Agent Call-Stacking Problem — samcharrington · 2026-11-18
- New Blog Post Explains TorchTPUs and How They Differ From Prior TPU Approaches — algo_diver · 2026-11-18
- Nvidia reportedly delayed its 10-Q filing by weeks, an extraordinary rarity — GaryMarcus · 2026-11-18
- TSMC starts checking customer balance sheets, demands cash to lock in capacity — SumitGup · 2026-11-18
- Cloudflare Workers static assets are free: 25MB per file, 20k files per version — ezshine · 2026-11-18