Analyzing Residual Write-back: Why Use 2 x Sigmoid for Initialization?
stochasticchasm · x · 2026-08-28
A technical analysis of a model's residual write-back operation suggests that the coefficient 2 is for initialization dynamics: since sigmoid(0)=0.5 and init is zero-centered, 2 x sigmoid is needed for normal scale writeback. The discussion also touches upon the absence of \overline{R} and the importance of Group-RMSNorm.
Related event: Residual Write-Back Design and AttnRes Depth Debate Draw Attention(2 posts)→
More from Research
- Michigan Robotics Rounds Up Its Papers and Workshops for IROS 2026 — doctorBobG · 2026-08-28
- Mouse Brain Connectome Cost Drops to $100M; Human at $1B — juanbenet · 2026-08-28
- Testing Muon Optimizer: Smoother Gradients and Stable Residual Maxima — stochasticchasm · 2026-08-28
- First open model adopts per-head orthogonalization following Kimi and GLM — stochasticchasm · 2026-08-28
- Integrity Bench: A New Benchmark to Measure Model Overconfidence — Acne_Discord · 2026-08-28
- Discussion on Classic Moonlight Scaling and Polar Express Orthogonalization — stochasticchasm · 2026-08-28