Hashimoto's harder quiz: does mixing residual states hurt even after LR retuning?
tatsu_hashimoto · x · 2026-10-14
Hashimoto's second prediction quiz: modify an 8-layer prenorm Transformer so that, from the second block onward, the residual update x{i+1}=xi+fi(xi) becomes x{i+1}=0.5xi+0.5x{i−1}+fi(xi). Both models train on the same 614.4M-token budget, and to avoid penalizing the variant for LR sensitivity, the best final val loss over six peak LRs (1e−4 to 3e−2) is taken for each. Predict Δ = Lmixed − Lstandard: does it help or hurt, and by roughly how much? The exact diff is provided for reproduction.
More from Research
- Surge launches sudo L7, a benchmark measuring staff-level engineering judgment in real codebases — echen · 2026-10-14
- Eterna community event covers RNA inverse design and 3D aptamer-ligand prediction — chaitjo · 2026-10-14
- GLOW: hybrid GNN + LLM question answering over incomplete knowledge graphs — burkov · 2026-10-14
- Google's AMIE hits The Lancet, matching doctors' final diagnoses 90% of the time — SumitGup · 2026-10-14
- MIT research shows neural networks can self-organize their own learning machinery — nordicinst · 2026-10-14
- Economist compiles 100+ free social science datasets on GitHub — _akpiper · 2026-10-14