SYNTH team's jspace analysis: Baguettotron layer topology radically differs from Gemma-270m
voooooogel · x · 2026-10-17
In an analysis left out of the SYNTH paper, Dorialexander shared jspace results for Baguettotron (350M, 80L): its commit zone topology differs radically from Gemma-3-270m (17L blocky reference), with very high feature connectivity. CKA heatmaps show a crystallization median around layer 60 and a smoother progression versus Gemma's blocky layering. The authors tentatively attribute this to synth pretraining rather than architecture depth.
More from Research
- Model tweaks barely move MMLU but easily break generative tasks — aryaman2020 · 2026-10-17
- sika-agent: open RL environment trains AI agents to catch MoMo scams in Twi and English — emeka_boris · 2026-10-17
- PKU & Samsung: slow model coaching lifts fast agent's live game score from 13.51 to 24.19 — rohanpaul_ai · 2026-10-17
- CoPE adds one forecasting loss to GRPO for 2x sample efficiency in agent RL — youjiaxuan · 2026-10-17
- LeanGraphSearch mines Mathlib dependency graphs, lifting proof success to 81% — kaiwei_chang · 2026-10-17
- rStation pools spare agent capacity from volunteers to tackle open research problems — progenitor414 · 2026-10-17