Gemma Successfully Resumes Training After Adding Layers
Desperate-Sir-5088 · reddit · 2026-07-12
The author shared an update on their second experiment "adding layers" to a Gemma model, aiming to prove that heavily modifying an already fine-tuned model won't necessarily break it; it can "recover and learn new layers."
Three key takeaways:
- New layers can't start out doing nothing. The first attempt failed because initializing the new layers as near-identity mappings meant they received almost no learning signal.
- Insertion position is critical. They switched to placing new layers in more stable locations rather than just wedging them into the original architecture.
- Don't mess with layer scaling parameters. Averaging these "volume knobs" in the first attempt degraded output quality significantly; preserving original settings worked much better this time.
Training happened in two stages: first, freezing the original model to train only the new layers so they could "find their footing"; then unfreezing the entire model for joint fine-tuning to ensure overall convergence. The author emphasizes the goal wasn't to beat the original model, but to validate that recovery is possible after major surgery on a fine-tuned model.
More from Research
- FoundationalGeo at ECCV 2026 highlights focal length gap in geo datasets — ducha_aiki · 2026-09-11
- DCVC backs new startup Humans& from gharik and Zelikman, launching Persimmon — gharik · 2026-09-11
- Researcher bootstraps from fly connectome to build increasingly intelligent connectomes — airkatakana · 2026-09-11
- CellFluxRL: RL-based biological grounding for virtual cell models, submitted to ECCV 2026 — Prof_Lundberg · 2026-09-11
- PiPNN nearest-neighbor search wins three awards, up to 78x faster index building — khademinori · 2026-09-11
- Steerable Visual Representations Presented as ICML Long Oral — y_m_asano · 2026-09-11