Beyond LORAs: soft prompts and GPU-maxing agent swarms proposed as inference cost paradigms
JoshPurtell · x · 2026-11-18
JoshPurtell outlines three paths for LLM inference cost: doing LORA-based inference properly (increasingly unlikely), exploring paradigms like soft prompts / KV (a relevant repo seems undecided but shows promising early results), or using agent swarms to max-load single-tenant GPUs. A reply notes soft-prompts haven't been heard about since 2022 but loves the direction — with autoresearch agents able to push arbitrary setups to SOTA (resin, deepseek-ultrafast), there's no excuse not to have zero-penalty adapted inference regarding batching/interactivity.
Related event: Developers Explore New Paths Beyond LORA for Cheaper LLM Inference(2 posts)→
More from Infra
- 594B model now runs on two DGX Sparks at 37.9 tok/s; drafter boosts 27B to 78 tok/s — AIFlow_ML · 2026-11-18
- Why Cheaper Tokens Can Still Mean a Bigger AI Bill: The Agent Call-Stacking Problem — samcharrington · 2026-11-18
- New Blog Post Explains TorchTPUs and How They Differ From Prior TPU Approaches — algo_diver · 2026-11-18
- Nvidia reportedly delayed its 10-Q filing by weeks, an extraordinary rarity — GaryMarcus · 2026-11-18
- TSMC starts checking customer balance sheets, demands cash to lock in capacity — SumitGup · 2026-11-18
- Cloudflare Workers static assets are free: 25MB per file, 20k files per version — ezshine · 2026-11-18