16 NVMe drives in RAID 0 match DDR5 bandwidth for LLMs
carrigmat · x · 2026-08-27
Challenging the notion that NVMe can't compete with RAM for LLM inference speed, the author proposes a cost-effective local deployment strategy:
- Bandwidth Math: A single Gen5 NVMe drive offers 13GB/s. A 16-drive RAID 0 array achieves 208GB/s, equivalent to 4-channel DDR5.
- Storage Strategy: Most model weights stay off RAM. Only 'hot' weights reside in memory, while the majority—intermittently accessed 'experts' in MoE models—remain on NVMe.
- Drive Longevity: Relying on modern SSDs' virtually infinite read lifespan. Weights are copied once and then strictly read, avoiding wear-out issues.
The approach aims to build a storage pool with near-memory bandwidth using consumer hardware to run massive models.
More from Infra
- Prefix Sliding enables efficient test-time scaling by cutting memory costs — Niklas Muennighoff · 2026-08-27
- LightningAI offers instant H100 access on its self-owned AI cloud — LightningAI · 2026-08-27
- Cohere releases Parse 5, claiming best price-performance for enterprise document parsing — irombie · 2026-08-27
- M7 Ultra may feature native FP8, potentially boosting GLM 5.3-flash performance — Brilliant-Hall1387 · 2026-08-27
- ChronoScale announces 50MW NVIDIA GB300 deployment with Microsoft for AI inference — r_jegaa · 2026-08-27
- Engineering Win: mxfp8 x mxfp4 Matmul Outperforms Standard mxfp8 — zephyr_z9 · 2026-08-27