New report details per-dimension forgetting, selective depth retrieval, and a 16-of-896 MoE
Ahmad_Al_Dahle · x · 2026-07-21
A reply highlights three architectural ideas in a new technical report:
- KDA replaces Gated DeltaNet’s single scalar decay with learned per-dimension forgetting.
- AttnRes selectively retrieves across depth instead of weighting every residual layer equally.
- LatentMoE activates only 16 of 896 experts, with routing balanced by quantiles of router scores.
The post frames these as fixes for common one-size-fits-all design choices in linear attention, residual weighting, and expert balancing.
Related event: Moonshot Releases 2.8T Open-Weights Model Kimi K3(14 posts)→
More from Models
- ChatGPT keeps telling people 404 Media is tech support for big corporations — 404 Media · 2026-10-26
- Claude's /show-me visualizes compressor auto makeup gain like a pro audio teacher — teropa · 2026-10-26
- ChatGPT keeps telling people 404 Media can fix their tech support problems — 404 Media · 2026-10-26
- Fern's "context is intelligence" thesis: prediction keeps improving out to 10.2M tokens — bneyshabur · 2026-10-26
- Mirendil launches Fern, a from-scratch pretrained model with up to 10M-token context — bneyshabur · 2026-10-26
- Users spot a dedicated translation UI appearing in Claude — BLCNYY · 2026-10-26