Running Multiple LLM Slots on One 7900 XTX Boosts Total Tokens/Sec
c2btw · reddit · 2026-11-08
作者在 7900 XTX 上用 HIP/ROCm 跑 Qwen 3 27B 级模型时发现,同一 GPU 上同时开多个推理 slot 的总 tokens/s 显著高于单 slot。他提出两个待解问题:
- 显存与上下文:slot 越多每个分到的上下文越小;系统提示词之间可能有大量重叠,但共享 KV 不好做,因为他的系统提示是链接到文件的极简结构,模型只读取与自身任务相关的部分。
- 任务可并行性:很多任务难以拆成互不依赖的并行子任务。他设想用共享上下文或消息板来协调,并尝试一个实验性架构:先用一个"编排" slot 把 prompt 拆成多个子任务分发给其他 slot,完成后编排 slot 自杀释放显存,还可接一个决策模型判断任务是否值得拆分。
帖子是求证/求助性质,但展示了本地多实例并行推理的实测收益,对本地部署玩家有参考价值。
More from coding & agent
- Baseten Loops goes GA with serverless SFT/RL training, per-token pricing — baseten · 2026-11-08
- Software engineer asks how to bring AI agents into his family's retail business — FIREATWlLL · 2026-11-08
- Where do your agents run when the laptop closes? 4 hosting options compared — michaelTM_ai · 2026-11-08
- Where do browser agents actually break on the real web? Login walls, popups, or worse — lucy_yarntra007 · 2026-11-08
- Anthropic's 40-min AI-native SDLC playbook: code is no longer the bottleneck — blaizedsouza · 2026-11-08
- The software factory is the next extinct wrapper: coding agents will absorb orchestration value in 3-6 months — blaizedsouza · 2026-11-08