Running Qwen 3.8 27B full-context on a 12GB laptop: two llama.cpp configs
PyaesoneP · reddit · 2026-08-19
After buying a gaming laptop (RTX 5070 Ti Mobile 12GB, Core Ultra 9 275HX, 32GB DDR5), the author found it GPU-poor for local LLMs—but still got Unsloth Qwen 3.8 27B UD Q4KXL running via llama.cpp + CUDA with full context and q80 KV cache, at 1.5–5 t/s. They share two tuned configs: (1) an agentic coding setup with OpenCode, -ctx 262144, FFN layers offloaded to CPU, flash attention, q8 KV cache, and MTP speculative decoding—slow (1.5 t/s at 180K context) but with enough context to hold the big picture; it's the first local model they trust on real (personal) projects. (2) A speed-prioritized personal assistant config with Hermes Agent (9–11.5 t/s, 98K context) that also fixed instruction-following on SOUL.md/MEMORY.md/USER.md rules the previous MoE model frequently ignored. Tensor offloading, credited to community posts, made a surprisingly big difference.
More from coding & agent
- Opus 5 picks materials and generates poses for robot configurator — freelerobot · 2026-08-19
- Claude Code Skill Automates Home Assistant Configuration & Dashboards — tom_doerr · 2026-08-19
- Luthn: Open-source local agent memory layer with access control and auditing — Illustrious_Tell_741 · 2026-08-19
- Brave vs Google Search API for AI Agents: The 2026 Enterprise Guide — Ok_pettech · 2026-08-19
- Window Assassin: Tray tool to kill processes hogging 1+ GB of VRAM — b2kdaman · 2026-08-19
- User report: Qwen 2.5 72B struggles with agentic coding tasks vs DeepSeek/Claude — BuahahaXD · 2026-08-19