Running Local Inference on a Laptop with an eGPU
jupiterbjy · reddit · 2026-07-19
The author shares a budget-friendly mobile local inference setup, tweaking llama.cpp parameters on a laptop to run Qwen3.6 30B A3B on limited hardware.
Key takeaways from the experience include:
- Adjusting llama.cpp parameters allows low-end hardware to run models effectively.
- The newly added built-in Web UI for llama.cpp is very lightweight, with an initial load of only about 3MB of JS/CSS.
- Current tool calling features lack a sandbox, which the author explicitly warns about, noting it's "interesting but unsafe."
- Screenshots show the laptop + eGPU + local inference monitoring dashboard, emphasizing the hands-on deployment experience.
Overall, it's a classic local deployment / edge inference tinkering post, focusing on engineering practice rather than the model itself.
More from Infra
- Resin lets an agent build standalone LLM inference engines; vLLM and SGLang optimize DeepSeek kernels differently — AccBalanced · 2026-10-27
- Local benchmark: Flint matches Qwen3.6 quality at 50/52 while running 3.6x faster on M4 Pro — RichardSocher · 2026-10-27
- Alchemy adds effect/Workflow support across Durable Objects, Workflows and Lambda Durable Functions — samgoodwin89 · 2026-10-27
- Alchemy's Lambda Web Functions run plain Node web servers on Lambda, no OpenNext needed — samgoodwin89 · 2026-10-27
- Foldkit Adds SSR Support With a Few Lines of Config on Cloudflare and AWS — samgoodwin89 · 2026-10-27
- Alchemy Cuts Cloudflare Bindings CPU Time by Up to 166× Via Profiling — samgoodwin89 · 2026-10-27