llama.cpp PR adds CPU offload for dense model FFN layers
pmttyji · reddit · 2026-08-19
A new PR for llama.cpp proposes adding --n-cpu-ffn for dense models, similar to the existing MoE option. This offloads FFN layers to the CPU, enabling larger dense models (like Qwen 3.8 27B) to run on VRAM-constrained devices (e.g., 16GB). Benchmarks show running the 27B model at Q4KM with 130k context and 20 t/s.
More from Infra
- FT warns of 'Darwin moment' for AI borrowers as CoreWeave bonds sink to distress levels — firstadopter · 2026-10-16
- Phone as interface, home servers as the lab: a two-workflow local AI setup — EAccelerate_42 · 2026-10-16
- "A datacenter is a bank": CoreWeave drama revives calls for算力 supervision — arthurcolle · 2026-10-16
- DOT-MoE: converting dense models into sparse MoE via optimal transport — burkov · 2026-10-16
- Stanford CS336: A 7B model needs 14GB for weights but 84GB to train — stanfordnlp · 2026-10-16
- New NVFP4 trick: compressing block scales saves 45-48% per expert matrix, bit-exact — TheZachMueller · 2026-10-16