SiliconBench: Speed, Memory, and Fidelity for LLM Serving on Unified-Memory Desktops
Ranran Haoran Zhang, Aysa Xuemo Fan, David Munhá Correia, Alex Cheema, Rui Zhang
cs.AR, cs.DC, eess.SY
2026-09-13
On a 64GB M5 Pro, vllm-metal more than doubles throughput from c=1 to 16; only llama.cpp, vllm-metal, and omlx pass SiliconBench’s three audit gates.
Local LLMs on Macs are no longer a niche. Ollama, llama.cpp, and LM Studio see hundreds of thousands of installs a year; mlxlm does 1.45 million PyPI downloads a month. Unified memory puts weights, KV cache, the browser, and the IDE in one RAM pool. A speed-only ranking misses two things: whether 4 to 16 parallel agent calls stall the scheduler, and how much headroom remains once inference sits near capacity. The same weights can also drift in task score versus an NVIDIA reference.
Most consumer benches time one or two stacks. SiliconBench runs nine Apple Silicon engines on matched chat and agent loads, adds memory traces and a classification fidelity check, and uses CUDA siblings on DGX Spark as a reference.
Three reading criteria: can the stack serve new architectures under concurrency (D1), does it keep unified-memory discipline (D2), and does multi-node scaling pay (D3).
The nine engines differ in packed versus padded batches, paged versus contiguous KV, whether prefill and decode share a forward pass, and whether memory is a hard cap or a Metal hint: llama.cpp, ollama, mlxlm, vllm-metal, vllm-mlx, omlx, sglang, hftransformers, mistral.rs.
The main audit runs Qwen3-0.6B BF16 on one 64GB M5 Pro (macOS 26.6) at concurrency 1/8/16. Chat prompts come from OpenOrca and CNN/DailyMail; agent prompts from BFCL v3, Hermes, and ClawsBench; 100 each. Agent contexts span roughly 0.3K–8.9K tokens. Qwen3.5-0.8B and Gemma-4-E4B-it probe newer architectures. Fidelity is weighted F1 on GMRID (N=1146, 8-class supply-chain incidents) against vLLM on an A100. A follow-up uses 4-bit Qwen3.8-27B and Qwen3.6-35B-A3B. The two-node study uses Qwen3.5-35B-A3B 8-bit on two 64GB M4 Pro minis over Thunderbolt 5 RDMA.
The snapshot cadence is designed around a two-week loop: a Claude Code agent patches adapters, humans review, then official numbers rerun. One MLX bump broke three stacks at once. An ollama bare-GGUF import dropped ChatML delimiters and caused about 92% parse failures on the classification task.
On Qwen3-0.6B, vllm-metal and hftransformers both advertise packed queries, paged KV, and mixed steps, yet chat throughput scales 3.65× versus 1.23× from c=1 to c=16. vllm-metal scales 2.71× on agent and is the only stack above 2× on both splits. Padded or serial-prefill paths gain less than 30%, regress, or fail by chat c=16. At chat c=16, vllm-metal and ollama finish within 1%; on agent, vllm-metal leads by 32% with 219 ms median TTFT. sglang admits work in 655 ms, but agent throughput falls 80.6 → 45.3 tok/s.
Six of nine complete at least 90/100 at every concurrency on both splits. mistral.rs degrades at c=8 and crashes at c=16; mlxlm falls to 2/100 at agent c=16; hftransformers misses the 1-hour agent cap. Only five stacks serve both Qwen3.5 and Gemma 4.
vllm-metal’s agent peak memory stays about 34.7–34.9GB as load rises; llama.cpp is similarly flat. ollama occupies 94–97% of Metal’s advisory working set from c=1. mistral.rs hits 59.3GB at agent c=8 and finishes 13/100. sglang and vllm-mlx declare budgets and complete all 600 requests while system memory approaches physical RAM and throughput falls. mlxlm nearly triples its chat footprint for a 7% throughput gain. A padding demo is harsher: three concurrent prompts of 30K+5K+10 tokens waste 61% of query/KV on padding; a 32GB M1 Pro hits page compression and 2.5× wall time, while 64GB barely pays.
On fidelity, six of eight Qwen3-0.6B stacks sit within about 1.5 points of the reference on both 0-shot and 5-shot. ollama’s 0-shot F1 is 0.4173 versus the reference 0.4094, but 5-shot stays at 0.4462 while peers reach 0.73, about 30 points behind. vllm-mlx is 5.0 points low at 0-shot. All five measured stacks on the newer models sit within 1.4 points.
Two-node decode: EXO 66.8 tok/s alone, mlxlm 49.3, llama.cpp 40.0. Tensor parallelism over RDMA takes EXO to 87.0 (1.30–1.37×) and mlxlm to 63.3 (1.28–1.32×). llama.cpp’s TCP pipeline parallelism drops to 32.4 (0.80–0.84×).
Only llama.cpp, vllm-metal, and omlx pass all three gates: ≥90/100 completion at every level, reference-band fidelity, and all three model families. CUDA vLLM and SGLang scale 4.3–7.7× on the same prompts with flat TTFT, so the Metal side still has scheduling headroom.
Picking a Mac serving stack by tok/s alone makes ollama and vllm-metal look tied on chat. Add agent concurrency, memory headroom, and 5-shot templates, and three stacks remain usable. An “explicit memory budget” does not mean the process stays small. Packed+paged+mixed steps do not guarantee scaling; hftransformers is the counterexample.
For local agents the sharper rule is: prefill has to share a forward pass with in-flight decode, or first-token latency blows up on long prompts under load. Do not default to pipeline parallelism over TCP; on these two minis it is slower than one machine.
The main audit is one M5 Pro and small models. Larger-model and two-node numbers use later builds and different quantization, so they are not a drop-in comparison with the 0.6B table. Fidelity is one classification task with no run-to-run variance. The 35B MoE fits on one box, so the two-node study measures interconnect efficiency, not capacity unlocking. Agent prompts top out near 9K tokens; 10K+ context and 1K+ generation are out of scope. Some of mlxlm’s 70/100 “failures” are valid tool calls with empty content, which biases the completion metric.
Snapshots rot. The paper leans on a maintainer agent for that; readers should still treat an August 2026 gate as a dated ranking.