A 13B model ran on a no-GPU PC by paging weights from SSD via llama.cpp
ID_R_McGregor · x · 2026-07-27
Brian Roemmele says he completed a stress test of flash-backed inference for a 13B model on ordinary consumer hardware with no discrete GPU.
- Setup: older Ryzen 5 desktop, 24 GB DDR4 RAM, 1 TB PCIe 3.0 NVMe SSD.
- The model is quantized to Q4KM and loaded with llama.cpp using --mmap, so the weights live on SSD and the OS pages in only the active tensors and a small working set.
- Reported results:
- peak RAM: 15.4 GB
- average output speed: 7.1 tokens/sec after warm-up
- SSD bandwidth: 412 MB/s peak, then 180–220 MB/s steady during generation
- He says the model kept coherent multi-turn behavior and is now aiming for a full chat interface.
More from Infra
- Nvidia gets mocked as “the leading open-source AI company” while repo chart shows it ahead — AccBalanced · 2026-07-27
- $8 ESP32-S3 runs a 28.9M-parameter LLM fully offline at 9.5 tokens per second — yangyi · 2026-07-27
- YC talk on BCI x AI says infrastructure is what really determines speed — garrytan · 2026-07-27
- llama.cpp warns that GGUFs made before a recent change must be regenerated — EconomySerious · 2026-07-27
- RTX 5090 local tests show Qwen Q6 can drop to 15 tok/s at 80k context — LFAdvice7984 · 2026-07-27
- Surprising Ubuntu Setup: NVIDIA 5090 PC Becomes the Easiest AI Rig — _xjdr · 2026-07-27