Could cross-stream mixing only be needed in the first half of the network? A look at mHC
teortaxesTex · x · 2026-10-28
teortaxesTex speculates that networks may need multiple streams throughout, but cross-stream mixing only in the first half (20 layers). Given mHC's competitiveness with alternatives, he argues late residual mixing may not be very useful — possibly one reason a model's V4.1 is so 'shallow'. He also notes a Grok translate failure mode: it dropped the Arxiv link.
More from Research
- New LALA benchmark: 236 humanities questions suggest more post-training can make LLMs dumber — JasonBotterill · 2026-10-28
- One-figure paper: the best optimizer depends on batch size, SOAP and Shampoo swap ranks — ricklamers · 2026-10-28
- Zero-training PCA head lifts 577M EmbeddingGemma-2 to #1 on MTEB audio, beating 9B models — JeanMichelRanu · 2026-10-28
- OpenAI's thousands of AI agents compress months of neurodegeneration research into 32 minutes — Dr_Singularity · 2026-10-28
- Self-distillation explained: how a model teaches itself with richer token-level training signals — arpit_bhayani · 2026-10-28
- Preclinical data lifts drug-response prediction in 27 of 38 compounds — david_van_dijk · 2026-10-28