Optimizing Qwen3.8-27B on MI300X boosts speed by 59%
cheptsov · reddit · 2026-08-21
dstack demonstrates optimizing Qwen3.8-27B inference on a single AMD MI300X using their open-source toolkit. By linking optimization sessions and applying source-level patches to SGLang's AITER attention backend, they increased inference speed from 311 to 495 tok/s (+59%). The optimized setup supports a 1M context with p50 TTFT under 1.5s and handles four concurrent users (10k in / 1.5k out). The result is a portable preset deployable on any AMD cloud, Kubernetes cluster, or bare-metal fleet.
More from Infra
- Haight Street Compute Market: a one-day trading floor for GPU deals in SF — ctjlewis · 2026-10-16
- Temporal Serverless Workers on Cloud Run: queue-driven autoscaling, scale to zero — rseroter · 2026-10-16
- LMA brings day-0 model support to TPU for the first time, spanning NVIDIA, AMD, Intel via SGLang — ying11231 · 2026-10-16
- Semtech's analyst-day slide cleanly explains the four signal-integrity impairments — BenBajarin · 2026-10-16
- Fortytwo Engine boosts Qwen3.8-27B accuracy 7% and cuts tokens 25% via Swarm Inference — rohanpaul_ai · 2026-10-16
- Berkeley scholars: concentrated chip supply chain makes a hardwired frontier AI pause feasible — wfithian · 2026-10-16