Cut LLM hardware costs with ROM-based weight storage
sirzerp · reddit · 2026-08-18
Michael Snyder proposes a new architecture to lower LLM deployment costs: "Frozen Weights, Fast ROM".
Core Idea:
Deployed model weights are read-only, yet stored in expensive, scarce HBM/DRAM. The paper suggests burning base weights into cheap, high-density Mask ROM (like game cartridges), keeping only a tiny LoRA-style diff in dynamic memory (DRAM) for fine-tuning and updates.
Rebutting "ROM is too slow":
The paper argues that ROM's speed bottleneck lies in the interface and packaging, not the storage element itself. Built with modern interfaces, ROM can run at RAM speeds by construction, and—free of capacitors, refresh, and write paths—it is denser and lower-power per die.
Related Work: This architecture combines existing technologies like LoRA (diff mechanism), Apple's Flash Streaming, and Taalas's hardwired-LLM ASIC (acquired by AMD).
More from Infra
- Logs show OpenAI/Anthropic bots not crawling llms.txt yet — gaganghotra_ · 2026-08-18
- M4 Pro achieves 8ms realtime inference for monocular depth model — fofrAI · 2026-08-18
- Cloudflare Durable Objects gets a deployments tab showing real traffic per version — ritakozlov · 2026-08-18
- Magnitude Catalog Profiles Hardware and Recommends Models — Dan_Jeffries1 · 2026-08-18
- Local vs. Sovereign AI: Where Is the Industry Drawing the Line? — rio_ARC · 2026-08-18
- City2Graph library turns geospatial data into spatial graphs — tom_doerr · 2026-08-18