Running Local Inference on a Laptop with an eGPU
jupiterbjy · reddit · 2026-07-19
The author shares a budget-friendly mobile local inference setup, tweaking llama.cpp parameters on a laptop to run Qwen3.6 30B A3B on limited hardware.
Key takeaways from the experience include:
- Adjusting llama.cpp parameters allows low-end hardware to run models effectively.
- The newly added built-in Web UI for llama.cpp is very lightweight, with an initial load of only about 3MB of JS/CSS.
- Current tool calling features lack a sandbox, which the author explicitly warns about, noting it's "interesting but unsafe."
- Screenshots show the laptop + eGPU + local inference monitoring dashboard, emphasizing the hands-on deployment experience.
Overall, it's a classic local deployment / edge inference tinkering post, focusing on engineering practice rather than the model itself.
More from Infra
- AI Infra Meme: While Countries Need TW-Scale Power for Datacenters, People Argue About Lightbulbs — mgostIH · 2026-10-27
- Legora runs 2,000 parallel agent evals on Modal Sandboxes for legal AI — blelbach · 2026-10-27
- Cloud release velocity scores: AWS 95, Azure 90, Google Cloud 82 — DavidLinthicum · 2026-10-27
- Selling a 5090 to scale local inference: one business weighs 5 hardware paths — Anarchaotic · 2026-10-27
- AWS tutorial: fleet-wide edge CV model lifecycle with SageMaker SDK v3, ~$0.07 per run — AWS ML Blog · 2026-10-27
- Qwen3.8-27B hits 40 tok/s on Jetson Thor with INT4 + MTP, accuracy beats FP16 — No-Dragonfly6246 · 2026-10-27