Hacking llama.cpp to hot-swap knowledge into Qwen's Ngram PLE table
ortegaalfredo · reddit · 2026-09-03
- Developer ortegaalfredo repurposes the Ngram PLE (prediction lookup) table in the new Qwen architecture as a long-term knowledge store: a patched llama.cpp updates the table in memory on every prompt, letting you hot-swap knowledge without reloading the model.
- Caveats: embeddings are injected early in the layers, so output control is imprecise, though simple modifications can still steer generation.
- Two repos released: a modified llama.cpp-NLTM and an ngram-knowledge-injector that compiles table patches.
- Requires memory-mapped PLE (llama.cpp default) and was only tested with q8 quantization, so it needs substantial RAM. Author suggests this could evolve into a low-cost training approach for models with instantly swappable long-term memory.
More from coding & agent
- antirez: Zero Out Steering When Possible—It Always Adds Distortion — antirez · 2026-09-03
- antirez Releases Steering Vector to Bypass DS4F Refusals in DwarfStar — antirez · 2026-09-03
- MiniMax H3 video editing and mask inpainting in ComfyUI on low VRAM — Maleficent-Tell-2718 · 2026-09-03
- ChatGPT Desktop's local Work mode can drive your scanner and archive docs — ___Patrice___ · 2026-09-03
- PointCloud Puzzle: why vibecoding still can't replace design taste — teortaxesTex · 2026-09-03
- Open-source TrueForge harness matches Claude managed agents' accuracy with 63% fewer tokens — Background-Job-862 · 2026-09-03