Random Engram Reads Kill SSDs: NVLink-Backed CPU Memory and Redis Eyed for Sparse Memory Lookup
bookwormengr · x · 2026-09-11
A technical X thread on serving sparse lookup-based memory (Engram) at scale: the author notes Hopper lacks NVL72 racks (the diagram was illustrative), and argues Engram's highly random reads make SSDs a poor fit for server-class use. Reading the Engram table from CPU memory over NVLink lets any of 72 GPUs running the model backbone access a single copy with low latency; an in-memory store like Redis could push lookup latency to single-digit milliseconds.
Related event: Engram's random reads strain SSDs, engineers debate in-memory solutions(2 posts)→
More from Infra
- MySQL recursive CTEs time out at 500M rows; C# BFS answers in 2.7ms vs NebulaGraph 23.6ms — JeremyCMorgan · 2026-10-28
- Token demand on OpenRouter grew 27x to 165T weekly, open models now serve 72% — FinanceYF5 · 2026-10-28
- AI's unusual bet: buying compute for years, selling intelligence by the token — vaibhavbetter · 2026-10-28
- OpenAI's Will Depue says compute locks in US primacy as 'end of human history' nears — willdepue · 2026-10-28
- Malaysia has overtaken Singapore in data center capacity, planned buildout 10x — bytebot · 2026-10-28
- Why Cloudflare Is the Best Platform to Build AI Agent Operating Systems — ritakozlov · 2026-10-28