Hacking llama.cpp to hot-swap knowledge into Qwen's Ngram PLE table
ortegaalfredo · reddit · 2026-09-03
- Developer ortegaalfredo repurposes the Ngram PLE (prediction lookup) table in the new Qwen architecture as a long-term knowledge store: a patched llama.cpp updates the table in memory on every prompt, letting you hot-swap knowledge without reloading the model.
- Caveats: embeddings are injected early in the layers, so output control is imprecise, though simple modifications can still steer generation.
- Two repos released: a modified llama.cpp-NLTM and an ngram-knowledge-injector that compiles table patches.
- Requires memory-mapped PLE (llama.cpp default) and was only tested with q8 quantization, so it needs substantial RAM. Author suggests this could evolve into a low-cost training approach for models with instantly swappable long-term memory.
More from coding & agent
- Open models below 4-bit struggle at self-iterating 3D asset generation, HF dev finds — mervenoyann · 2026-10-21
- Developer Moves AI Agents to a Dedicated Linux Workstation Using Incus System Containers — kevinkern · 2026-10-21
- Motion Launches a General Video Agent That Edits Footage and Mimics Any Reference Style — mhdfaran · 2026-10-21
- MCP Platform NitroStack Wins Product Hunt Product of the Day, Focuses on Retention — EstablishmentFun4373 · 2026-10-21
- CodeRabbit Ships 5,000 PRs a Week With a Tiny Team, Hires by Strategy-Game Hours — omojumiller · 2026-10-21
- LukeW maps the five UI problems every AI-native company faces with agents — LukeW · 2026-10-21