No Input/Output Optimized Params, and No Weight Decay on the LM Head
stochasticchasm · x · 2026-09-11
While examining a model architecture featuring a gigantic engram table, stochasticchasm initially thought it used no Adam optimizer at all, then corrected: some Adam-optimized parameters do remain — but the absence of any input/output optimized parameters is still remarkable. A follow-up in the same thread adds that there is no weight decay on the LM head either.
Related event: Leaked DeepSeek V4.1 Flash Tech Report Reveals KV Cache-Obsessed Flagship(44 posts)→
More from Models
- DeepSeek cut KV cache per token from 389KB to 890B in two years as open-closed gap debate rages — teortaxesTex · 2026-10-27
- Unverified: fable 5.5 speculated at 4-5T parameters to outscale 1-3T Opus — haider1 · 2026-10-27
- Kimi K3 Is 'Open' Until You Succeed: License Caps Free Use at $20M Revenue — alex_verem · 2026-10-27
- Screenshot shows 'GPT-6 Astra Ultra' in ChatGPT model picker — unverified leak or joke — justalexoki · 2026-10-27
- DeepSeek Safety Eval Finds "Over-Attribution": Model Mistakes Ordinary Tasks for Tests — Much_Boysenberry9124 · 2026-10-27
- Custom VQA benchmark under 48GB VRAM: Qwen3.8-Flash-Next leads, Qwen3.8-27B regressed — EmilPi · 2026-10-27