Splitting one 27B across two 3090s was slow: one model per GPU hits 243 tok/s combined
StewartalsopIII · x · 2026-11-06
A user shared local inference benchmarks: running a single 27B model sharded across two RTX 3090s yielded only 50-60 tok/s for months. Switching to one model per GPU — a fast Qwen 3.8 Flash-Next with 262K context on one card and a 27B with 128K context on the other — enabled six concurrent streams measured at a combined 243 tok/s.
More from Infra
- Yukon syncs tables near-real-time at 1M events/sec, built so agents can query what changed — testingcatalog · 2026-11-06
- Stanford's Joules-to-GDP measures the economic value AI produces per joule of energy — rohanpaul_ai · 2026-11-06
- GPU cloud Lambda rebrands from lambda.ai to lambda.com — TheZachMueller · 2026-11-06
- Texas data center with zero-water cooling and full power self-supply still rejected — GabGarrett · 2026-11-06
- OpenAI pledges local residents won't pay electricity costs of its AI data centers — Polymarket · 2026-11-06
- Reuters: 600,000 Nvidia H200 chips stuck in Hong Kong; 8-GPU server hits $900K on gray market — teortaxesTex · 2026-11-06