Qwen3.5-9B runs offline on a stock PS4 at 39 tok/s using its own MTP head
leonardo_alegria · reddit · 2026-11-14
Developer leonardoalegria's open-source project garlic-llm runs Qwen3.5-9B offline on a stock PS4's GPU at 31–39 tok/s. It talks, listens, and writes code — no cloud, no subscription, no Linux required.
Key techniques:
- Uses Qwen3.5's bundled MTP head for speculative decoding: it drafts up to two tokens and the full model verifies them in one pass, lifting throughput from 22 to 31–39 tok/s on the old GPU.
- Weights are re-packed into custom 4- and 3-bit formats to fit 4.5 GiB into the PS4's memory.
The code is open-sourced on GitHub, with weights downloadable from Hugging Face.
More from Infra
- Open-weights TTS model Tontaube V1.1 runs near real-time on a desktop CPU in 4-bit — EAVDR · 2026-11-14
- MiniMax H3 Image goes live on TensorScale, claimed 2.3x faster and 5x cheaper than fal at 4K — TerryTangYuan · 2026-11-14
- Li Bojie open-sources 'Understanding AI Infra' book, 7,500+ GitHub stars in weeks — udmrzn · 2026-11-14
- Prime Intellect jumps into top 10 third-party inference providers at 2.22T weekly tokens — Ronangmi · 2026-11-14
- LMSYS launches SGLang Consortium with NVIDIA, AMD, Intel, Google, Meta and more — NVIDIAAI · 2026-11-14
- US voters rejected data centers in 40 of 41 ballots; UK overrules councils to build them — thealexbanks · 2026-11-14