Ngram Module Experiments: Loss Not a Perfect Signal
nrehiew_ · x · 2026-08-27
Regarding the Ngram module (borrowed from DeepSeek Engram), ablations on layer placement showed no clear winner, leading to placement at layer 2 for CPU prefetching. Data suggests loss is not a perfect signal: downstream loss decreases with vocab size, but this does not consistently translate to better evals.
Related event: Qwen Details Training Recipes: Muon Optimizer and Residual Stream Findings(5 posts)→
More from Research
- U of T team mines 122,194 patient reviews across 45 Ontario hospitals with NLP to expose satisfaction and equity patterns — ishtiaqueSIA · 2026-10-16
- University of Toronto hiring AI postdoc: builders of interventions, not just benchmarkers, due Oct 30 — ishtiaqueSIA · 2026-10-16
- WorldMemArena Accepted at NeurIPS: Memory Storage ≠ Memory Use — xwang_lk · 2026-10-16
- Mech Interp Veteran Rethinks Alignment, Misgeneralization, CoT Oversight — akbirthko · 2026-10-16
- WEDGE paper generates training environments conditioned on navigation agent failures — _akhaliq · 2026-10-16
- RExBench: best coding agent completes only 33% of autonomous research extensions — najoungkim · 2026-10-16