llama.cpp prefill leaves CPU cores and memory bandwidth surprisingly idle
Dependent_Ad948 · reddit · 2026-07-30
llama.cpp CPU prompt processing appears to underuse cores and memory bandwidth
A user experimenting with CPU-only inference in mainline llama.cpp says prompt processing behaves very differently from token generation.
Observed difference
On a Xeon Ice Lake system with 24 physical cores and 8 populated memory channels:
- Inference/token generation pegs the assigned CPU cores and pushes memory bandwidth hard.
- Prompt processing/prefill mostly pins one core and uses far less overall CPU and DRAM bandwidth.
The author reports roughly:
- Inference: 2302% CPU, 31.1% MEM, and about 81,723 MB/s DRAM read throughput.
- Prompt processing: 625.2% CPU, 31.1% MEM, and about 15,917 MB/s DRAM read throughput.
What they want to understand
The goal is not raw speed, but to understand why prefill seems to leave so much compute and memory throughput unused, especially when some layers are offloaded from the GPU. The post asks what the user may be missing and what concepts or tuning knobs would explain the gap.
More from Infra
- Baseten Launches Model Labs Platform for Closed-Model Monetization — baseten · 2026-07-30
- Baseten Launches Model Labs Platform for Commercializing Closed Models — baseten · 2026-07-30
- Replacing Cloud Vision APIs Locally with Nvidia Nemotron on DGX Spark — JFPuget · 2026-07-30
- Laguna XS Breaks 140 TPS on Apple Machines with New FAST Mode — gajesh · 2026-07-30
- $50B+ in AI Data Center Leases Signed in July as Bitcoin Miners Pivot — abhiadesai · 2026-07-30
- 26B-Parameter Gemma 4 Runs on Mac with 2GB RAM via SSD Streaming — petrusenko_max · 2026-07-30