Running Local Inference on a Laptop with an eGPU
jupiterbjy · reddit · 2026-07-19
The author shares a budget-friendly mobile local inference setup, tweaking llama.cpp parameters on a laptop to run Qwen3.6 30B A3B on limited hardware.
Key takeaways from the experience include:
- Adjusting llama.cpp parameters allows low-end hardware to run models effectively.
- The newly added built-in Web UI for llama.cpp is very lightweight, with an initial load of only about 3MB of JS/CSS.
- Current tool calling features lack a sandbox, which the author explicitly warns about, noting it's "interesting but unsafe."
- Screenshots show the laptop + eGPU + local inference monitoring dashboard, emphasizing the hands-on deployment experience.
Overall, it's a classic local deployment / edge inference tinkering post, focusing on engineering practice rather than the model itself.
More from Infra
- LLM Serving Metrics Thread: Why TPOT and Uptime Make or Break User Experience — abhijithneil · 2026-09-11
- PlanetScale launches sharded Postgres: 768 servers acting as one, 1PB scale — dhruv2038 · 2026-09-11
- Can a 7900 XTX 24GB run Qwen locally? Reddit seeks ROCm tok/s benchmarks — thenomadexplorerlife · 2026-09-11
- RTK Terminal Compression Cuts Tokens but Leaves Your AI Coding Bill Unchanged — Bartaseth · 2026-09-11
- SF Compute founder: buying compute is 'an absolutely awful experience' right now — IgorCarron · 2026-09-11
- SmolVM open-sources persistent computer infrastructure for agents that outlive chat sessions — aniketmaurya · 2026-09-11