Open-source Mac Studio + RTX 5090 sidecar hits ~210 tok/s on Qwen3.8
GabGarrett · x · 2026-11-10
The open-source Apple NVIDIA Sidecar project (released Nov 8) connects an RTX 5090 in a Thunderbolt dock to a Mac Studio (M5 Max 64GB), running Qwen3.8 Flash-Next with NVIDIA's official FP4 weights at 210 tok/s single-stream decode — beating published numbers from other single-machine setups.
Reference points (methodologies vary, ballpark only):
- 1x DGX Spark 43 tok/s; Jetson AGX Thor 47; M4 Max peak 83
- RTX 5090 PC with 128GB RAM (NVFP4): 114-116; 72GB RTX PRO 5000: 159 median
Known limitations:
- 8K-prompt prefill 1,900 tok/s (good PC setups exceed 3,900)
- Single-session serving; total output stays 61 tok/s across 1-16 concurrent streams
- Thunderbolt link measured at 6.79 GB/s one-way, 5.53 reverse (rated 7.88)
- Drivers are the author's locally modified builds, not in the repo — reproducers must sort out drivers themselves
More from Infra
- Autonomous' desk-size datacenter: 2x RTX PRO 6000 from $26,100 — dee_hw · 2026-11-10
- Taobao MNN-Pocket-0.3B trends on Hugging Face: Qwen3-based on-device INT4 model — taobao-mnn · 2026-11-10
- Developer Asks: Has Anyone Tried Cloudflare's Auto Router for LLM Routing? — MartinGTobias · 2026-11-10
- Search and recsys are all deep transformer architectures now — outsiders have no idea — burny_tech · 2026-11-10
- Hetzner's eBPF-based hcnet network stack hits beta with 240k concurrent connections per server — DanielLockyer · 2026-11-10
- Cisco booked $9.3B in hyperscale AI orders, 4.5x YoY, and is betting on agent security — shashib · 2026-11-10