vLLM serves Qwen3.8-Flash FP4 on a Mac at ~40 tok/s for local agents
alfcnz · x · 2026-11-14
alfcnz shares his local agent setup: after running out of Claude and Codex tokens, he now serves nvidia/Qwen3.8-Flash-Next-FP4-Local on his own Mac via vLLM at 40 tokens/sec (screenshot: 0.16s first token, 2.2s prefill for 2k tokens, 42.6 tok/s generation).
The local agent generates learning lessons and a personal assistant on demand, having served 453.6k generated tokens, 61.5M prompt tokens and 592 requests so far — a working demo of self-hosted agent inference on a consumer Mac.
More from Infra
- Musk: Future Starship Flights Will Carry Over 50 Tb/s of Starlink Bandwidth to Orbit — elonmusk · 2026-11-14
- Chart resurfaces showing GitHub outages spiked after Microsoft acquisition — zacharynado · 2026-11-14
- Targon SDK update unifies VMs, bare metal and sandboxes in Python, Go and TypeScript — JosephJacks_ · 2026-11-14
- AWS Emulator Goes GA: 200+ Services Run Locally, Built Almost Entirely by AI Agents — iamfakhrealam · 2026-11-14
- FAA, Microsoft 365, Claude, Workday All Broke in 48 Hours — No Hackers Needed — r0ck3t23 · 2026-11-14
- llama.cpp ASR/TTS support roundup: Qwen3-ASR/TTS landing, audio output still in planning — Kahvana · 2026-11-14