Layer-by-layer analysis shows value residuals are actively used, peaking at layer 6 of Ripple 65B
DZhang50 · x · 2026-11-10
DZhang50 (from the model team) quotes remior's layer-by-layer analysis of value residuals in Ripple 65B, adding design rationale:
- Value residuals were added for a small CE win and, more importantly, to reduce gradient spikes at 8B 4xC scale
- remior plotted the value residual coefficient per layer: higher coefficient = more reliance on the residual, less on local hidden states
- Coefficients are far from zero: early layers rely on it surprisingly heavily, layer 6 builds its value projection only from input embeddings, and strength peaks at layer 6 then decays
A rare empirical check on whether a specific architectural component is actually doing useful work.
More from Research
- Frontier3DBench: 145 tasks test whether coding agents can replace 3D vision pipelines — Aiden_Tech_Ai · 2026-11-10
- JEPA is just a new name for old tricks: iBOT, data2vec and even CPC were basically I-JEPA — TimDarcet · 2026-11-10
- Reframe: NJU's Two-Stage Post-Training Framework Aligns Visual Agents' Observations, Beats Mix-RL and Vanilla MOPD — NJU · 2026-11-10
- Spatial Harness Grounds VLM Spatial Reasoning in Executable Physics-Engine World States — TMLR-Group-HF · 2026-11-10
- InstantHMR Runs Real-Time 3D Human Mesh Recovery on Phone NPUs: 5.5ms per Person, 12x Faster Than Peers — naturalpad · 2026-11-10
- 360-trial benchmark finds code knowledge graphs give coding agents no clear accuracy gain — JeremyCMorgan · 2026-11-10