vLLM adopts Strata-style trick: 51B-param Engram lookup table runs without residency in VRAM
solyarisoftware · x · 2026-11-01
vLLM now supports Engram, a conditional memory module (a 51B-parameter PLE n-gram lookup table in Qwen3.8-Flash-Next), using the trick Strata popularized: the table stays inside the original safetensors files instead of residing in VRAM or pinned RAM.
At each inference step, token IDs determine which table rows are needed; only those rows are fetched and copied into small GPU buffers for decoding. Engram hashes preceding tokens into large static embedding tables and gates the retrieved rows into the hidden state — like MoE scales capacity via conditional computation, Engram scales it via conditional memory, with per-token cost staying O(1). vLLM enables it automatically whenever a checkpoint declares Engram layers, with controls for memory footprint and multi-GPU topology.
More from Infra
- Musk claims xAI's capex efficiency is greatly underestimated — RachelVT42 · 2026-11-01
- AI bubble pops globally: China wipes out ¥23.5T, Nasdaq drops 3.46% in a day — Grady_Booch · 2026-11-01
- Strata engine underwhelms on 2× R9700: loses to llama.cpp despite higher t/s — Don_Reuter · 2026-11-01
- exo clusters your Macs into one AI machine to run frontier models locally — alex_verem · 2026-11-01
- 6 open source tools to run AI locally on your laptop, no cloud required — alex_verem · 2026-11-01
- Junction Launches Two-Week All-Expense-Paid Chip Fab Retreat in Bangalore to Fast-Track India's Semiconductor Talent — arpitingle · 2026-11-01