Hacking llama.cpp to hot-swap knowledge into Qwen's Ngram PLE table
ortegaalfredo · reddit · 2026-09-03
- Developer ortegaalfredo repurposes the Ngram PLE (prediction lookup) table in the new Qwen architecture as a long-term knowledge store: a patched llama.cpp updates the table in memory on every prompt, letting you hot-swap knowledge without reloading the model.
- Caveats: embeddings are injected early in the layers, so output control is imprecise, though simple modifications can still steer generation.
- Two repos released: a modified llama.cpp-NLTM and an ngram-knowledge-injector that compiles table patches.
- Requires memory-mapped PLE (llama.cpp default) and was only tested with q8 quantization, so it needs substantial RAM. Author suggests this could evolve into a low-cost training approach for models with instantly swappable long-term memory.
More from coding & agent
- Nous Research's Hermes now launches Cua Spaces for computer-use agents — alexcovo_eth · 2026-10-21
- Headlong from Laude's Slingshots batch is a microharness for persistent agents — lateinteraction · 2026-10-21
- Laude Institute's Slingshots Four ships 14 open-source projects on agent harnesses and open models — lateinteraction · 2026-10-21
- Skills vs MCP vs RAG vs Memory: the four knowledge layers production agents need — MaryamMiradi · 2026-10-21
- When customer service bans AI agents, one goes incognito to fight Xfinity — MoonL88537 · 2026-10-21
- Every.to publishes raw interviews so readers can agent-check quotes in its Codex story — every · 2026-10-21