Gemma Successfully Resumes Training After Adding Layers
Desperate-Sir-5088 · reddit · 2026-07-12
The author shared an update on their second experiment "adding layers" to a Gemma model, aiming to prove that heavily modifying an already fine-tuned model won't necessarily break it; it can "recover and learn new layers."
Three key takeaways:
- New layers can't start out doing nothing. The first attempt failed because initializing the new layers as near-identity mappings meant they received almost no learning signal.
- Insertion position is critical. They switched to placing new layers in more stable locations rather than just wedging them into the original architecture.
- Don't mess with layer scaling parameters. Averaging these "volume knobs" in the first attempt degraded output quality significantly; preserving original settings worked much better this time.
Training happened in two stages: first, freezing the original model to train only the new layers so they could "find their footing"; then unfreezing the entire model for joint fine-tuning to ensure overall convergence. The author emphasizes the goal wasn't to beat the original model, but to validate that recovery is possible after major surgery on a fine-tuned model.
More from Research
- Hermes Agent rewrite proposal applies RIA and Logic Bus rules — Promptmethus · 2026-07-22
- AllTheBacteria turns 2.44 million genomes into an AI-ready resource for new antibiotics — shae_mcl · 2026-07-22
- WeirdChat catalogs strange model behaviors from more than 100 million sampled responses — JacobSteinhardt · 2026-07-22
- New agentic benchmark shows AI managers escalate to coercion and fake success — Jasmine Brazilek · 2026-07-22
- Ai2’s Asta adds one-click handoff and self-checking deep paper search — allen_ai · 2026-07-22
- NVIDIA says physical AI starts in simulation with OpenUSD and synthetic data — MonaJalal_ · 2026-07-22