SiliconBench: speed, memory and fidelity of nine LLM engines on unified-memory desktops
PennState · hf · 2026-09-22
- SiliconBench evaluates nine LLM serving engines on unified-memory Apple Silicon desktops across speed, memory, and fidelity, covering chat and agent serving on Qwen3, Qwen3.5, and Gemma 4, with an NVIDIA reference for quality regression and DGX Spark as a complementary reference.
- Key findings:
- vllm-metal more than doubles throughput from concurrency 1 to 16 on Qwen3-0.6B; CUDA vLLM and SGLang scale better under concurrency.
- Explicit memory budgets don't guarantee headroom: two stacks complete every request while nearing physical memory limits.
- Newer architectures have narrower engine support; only three stacks pass all completion, fidelity, and model-coverage gates.
- vllm-metal's packed prefill-decode path keeps lower first-token latency than omlx under load.
- Two-machine tests: tensor parallelism over Thunderbolt RDMA scales; pipeline parallelism over TCP regresses.
- Benchmark code, per-run results, and maintenance journals are open-sourced.
More from Infra
- python-build-standalone enables full LTO for CPython 3.12+, modestly boosting runtime — charliermarsh · 2026-09-22
- Measured trade-offs of three REAP-pruned Qwen3.8-Flash-Next MLX builds on Apple Silicon — MensaProdigy · 2026-09-22
- Dev claims further-optimized DeepSeek V4 NVFP4 uses 190GB of 192GB VRAM — HankYeomans · 2026-09-22
- 'AWS made the industry soft': AI infra isn't mature enough to outsource the hard parts — mgill25 · 2026-09-22
- Grass network audited: 3M+ users, $32.1M revenue serving AI training data — Ronangmi · 2026-09-22
- SemiAnalysis: mapping MoE models onto inference hardware — zephyr_z9 · 2026-09-22