Gemma Successfully Resumes Training After Adding Layers
Desperate-Sir-5088 · reddit · 2026-07-12
The author shared an update on their second experiment "adding layers" to a Gemma model, aiming to prove that heavily modifying an already fine-tuned model won't necessarily break it; it can "recover and learn new layers."
Three key takeaways:
- New layers can't start out doing nothing. The first attempt failed because initializing the new layers as near-identity mappings meant they received almost no learning signal.
- Insertion position is critical. They switched to placing new layers in more stable locations rather than just wedging them into the original architecture.
- Don't mess with layer scaling parameters. Averaging these "volume knobs" in the first attempt degraded output quality significantly; preserving original settings worked much better this time.
Training happened in two stages: first, freezing the original model to train only the new layers so they could "find their footing"; then unfreezing the entire model for joint fine-tuning to ensure overall convergence. The author emphasizes the goal wasn't to beat the original model, but to validate that recovery is possible after major surgery on a fine-tuned model.
More from Research
- AI folks, stop saying "biology" when you mean drug development — KevinKaichuang · 2026-10-26
- ostris Pretrains 'Primal VAE': Aggressive 2x-Downsample, 1-Channel Latent Design — ostrisai · 2026-10-26
- ostris pre-training "Primal VAE": 1-channel latents match modern 8x VAEs via patching — ostrisai · 2026-10-26
- Shanghai AI Lab-led team releases 100-hour egocentric wearable dataset for humanoid training — ChuckDBrooks · 2026-10-26
- ConvFill: A 0.6B Talker Buys Time While a Large Reasoner Whispers the Real Answer — andimarafioti · 2026-10-26
- Flow Reasoning Models turn flow matching into iterative reasoners, hitting 99.5% on Sudoku-Extreme — alec_helbling · 2026-10-26