A 4GB laptop GPU can run a local Qwen agent stack, but RAG forces model swaps
maikerukonare · reddit · 2026-07-26
The author benchmarked a local AI agent workspace on a 4GB RTX 3050 Ti laptop GPU, using self-hosted Qwen models via Ollama with no cloud keys.
Key findings:
- qwen3.5:0.8b: 122 tok/s, about 1.4GB VRAM, fully on GPU
- qwen3.5:2b-q4KM: 96 tok/s, about 2.4GB VRAM, still fully on GPU
- qwen3.5:4b: 25 tok/s, about 3.4GB VRAM, with roughly a third spilling to CPU
- qwen3.5:9b: 8.6 tok/s, mostly on CPU, wants around 8GB
The 2B model is the sweet spot: it fits with about 1.3GB to spare and is fast enough for chat, tool calls, and vision. The post also breaks down a practical RAG setup on 4GB:
- the embedding model and chat model cannot stay resident together, so Ollama swaps them in and out
- embedding happens in a background worker so ingestion does not block chat
- a query incurs a reload cycle: embed the query, load the chat model, answer, then swap back
- if you want to avoid that cost, use a much smaller embedder or CPU embeddings
The author also notes where small models still fail:
- artifact generation often produces broken HTML unless switching to a coder-tuned model
- tool selection degrades as the tool set grows
Overall, it is a concrete report on what a full local agent workspace can and cannot do on 4GB VRAM.
More from coding & agent
- Cognition's SWE-2 uses a KKT duality argument in RL to shift the effort Pareto curve — YouJiacheng · 2026-09-11
- First-ever Three.js Conference lands in Paris, with a panel on AI-shortened design workflows — OdinLovis · 2026-09-11
- Data engineering, not agent frameworks, is the real bottleneck for enterprise AI agents — dhruv2038 · 2026-09-11
- RTK Terminal Compression Cuts Tokens but Leaves Your AI Coding Bill Unchanged — Bartaseth · 2026-09-11
- GPT-6 Astra beats Factorio with enemies in 44 in-game hours at ~$4,500 API cost — liminal_bardo · 2026-09-11
- Investment Analyst Asks How to Build a Claude-Based Diligence Agent Stack — Careless_Tie2286 · 2026-09-11