A 4GB laptop GPU can run a local Qwen agent stack, but RAG forces model swaps
maikerukonare · reddit · 2026-07-26
The author benchmarked a local AI agent workspace on a 4GB RTX 3050 Ti laptop GPU, using self-hosted Qwen models via Ollama with no cloud keys.
Key findings:
- qwen3.5:0.8b: 122 tok/s, about 1.4GB VRAM, fully on GPU
- qwen3.5:2b-q4KM: 96 tok/s, about 2.4GB VRAM, still fully on GPU
- qwen3.5:4b: 25 tok/s, about 3.4GB VRAM, with roughly a third spilling to CPU
- qwen3.5:9b: 8.6 tok/s, mostly on CPU, wants around 8GB
The 2B model is the sweet spot: it fits with about 1.3GB to spare and is fast enough for chat, tool calls, and vision. The post also breaks down a practical RAG setup on 4GB:
- the embedding model and chat model cannot stay resident together, so Ollama swaps them in and out
- embedding happens in a background worker so ingestion does not block chat
- a query incurs a reload cycle: embed the query, load the chat model, answer, then swap back
- if you want to avoid that cost, use a much smaller embedder or CPU embeddings
The author also notes where small models still fail:
- artifact generation often produces broken HTML unless switching to a coder-tuned model
- tool selection degrades as the tool set grows
Overall, it is a concrete report on what a full local agent workspace can and cannot do on 4GB VRAM.
More from coding & agent
- Most agent teams should tune the harness before touching model weights — hwchase17 · 2026-07-26
- Model migrations should prove behavior, not just API compatibility — hwchase17 · 2026-07-26
- Owning AI also means owning the data that powers it — hwchase17 · 2026-07-26
- Jeff Dean says context engineering beats raw model size at YC Startup School — ycombinator · 2026-07-26
- Terminal-Bench Score Reaches 60% Running on 56GB VRAM with Single RTX Pro 6000 — victormustar · 2026-07-26
- Graphiti hybrid search misfires on domain-specific memories, even when the LLM gets it right — Opposite_Courage_531 · 2026-07-26