Quiz: Double the Residual Stream from 4096 to 8192 — How Does Param Count Change?
Sauers_ · x · 2026-10-16
A quiz tweet for LLM practitioners: if you double the residual stream width from 4096 to 8192 while keeping the number of layers, MLPs, and attention heads unchanged, how does the total parameter count change? The post gives no answer itself, testing whether readers understand how params scale with dmodel across attention projections, MLPs, and embeddings.
More from Research
- 200 chess games suggest in-context learning doesn't work yet, experimenter says — akbirthko · 2026-10-16
- Mosaic: multi-session agentic RL with hierarchical advantage estimation — LARK-Lab · 2026-10-16
- Meituan LongCat's WorldCast benchmark: GPT-Image-2.5 tops at 72.8, open-source lags — meituan-longcat · 2026-10-16
- Recursive Environment Scaling tanks six models' success rates by 57–77 points — Lingxiao Jing · 2026-10-16
- Harvard's MPT: decentralized agent swarms learn self-orchestration via bidding — Harvard · 2026-10-16
- MIT Biology and Schwarzman College hiring tenure-track professor in computational biology with generative AI focus — anshulkundaje · 2026-10-16