One-line T5 fix left unmerged for 3 years: Hugging Face's closed PR on lm_head init
Birchlabs · x · 2026-10-21
Developer Birch-san notes Hugging Face never merged his one-line transformers PR (#26441) from 2023. It changes T5 lmhead weight init variance from 1 to the reciprocal of hiddendim: before, logits had variance hiddendim and initial cross-entropy loss of 110; after, variance 1 and initial loss 11, in line with expectations. Related to issue #16749; the PR remains closed and unmerged.
More from Fun
- Don't Trust AI on Electronics Wiring, Engineer Warns After UART Cable Mix-Up — kingslyj · 2026-10-21
- New zeroth-order speedrun optimizers sport names like Dust, Relay, Drift Leap — willcb · 2026-10-21
- Creative dev makes live Libraries website react when a real magnet is nearby — LinusEkenstam · 2026-10-21
- A CSS eye chart poster where every line's size is its own font-size, down to 0.5px — jh3yy · 2026-10-21
- AGI is basically a green lantern ring, per this comic meme — snowclipsed · 2026-10-21
- Dave Ramsey's 'get rich' secret, satirized: raise $100M and scatter $500K checks — StewartalsopIII · 2026-10-21