A 4GB laptop GPU can run a local Qwen agent stack, but RAG forces model swaps

maikerukonare · reddit · 2026-07-26

The author benchmarked a local AI agent workspace on a 4GB RTX 3050 Ti laptop GPU, using self-hosted Qwen models via Ollama with no cloud keys.

Key findings:

qwen3.5:0.8b: 122 tok/s, about 1.4GB VRAM, fully on GPU
qwen3.5:2b-q4KM: 96 tok/s, about 2.4GB VRAM, still fully on GPU
qwen3.5:4b: 25 tok/s, about 3.4GB VRAM, with roughly a third spilling to CPU
qwen3.5:9b: 8.6 tok/s, mostly on CPU, wants around 8GB

The 2B model is the sweet spot: it fits with about 1.3GB to spare and is fast enough for chat, tool calls, and vision. The post also breaks down a practical RAG setup on 4GB:

the embedding model and chat model cannot stay resident together, so Ollama swaps them in and out
embedding happens in a background worker so ingestion does not block chat
a query incurs a reload cycle: embed the query, load the chat model, answer, then swap back
if you want to avoid that cost, use a much smaller embedder or CPU embeddings

The author also notes where small models still fail:

artifact generation often produces broken HTML unless switching to a coder-tuned model
tool selection degrades as the tool set grows

Overall, it is a concrete report on what a full local agent workspace can and cannot do on 4GB VRAM.

Original post →

A 4GB laptop GPU can run a local Qwen agent stack, but RAG forces model swaps

More from coding & agent