Baidu's quaternion-based oHC residual connections cut BPB by 6.2σ but only delay stream homogenization
teortaxesTex · x · 2026-10-28
teortaxesTex highlights Baidu's paper on Orthogonal Hyper-Connections on SO(4) via quaternions (oHC): in theory it beats existing residual-stream schemes — keeping cross-stream communication unlike iHC, avoiding stream homogenization unlike mHC, while being simpler and faster.
Key results (3.9B model, 73B tokens):
- oHC cuts mean BPP across 16 benchmarks by 3.9% vs a single-stream residual baseline, a 6.2σ effect, beating mHC (1.8σ) and iHC (3.1σ);
- The quaternion pair is the cheapest orthogonal construction — 20.6× cheaper than the Sinkhorn–Knopp iteration it replaces, still 2.7× cheaper with a handwritten fused Triton kernel;
- But empirically it only delays homogenization by 3 layers of equivalent depth rather than preventing it — streams converge purely through shared updates.
This raises a deeper question: how much does all this mixer engineering matter in truly large and deep networks?
More from Research
- REGER paper adds explicit item relations to generative recommendation via semantic IDs — _reachsumit · 2026-10-29
- ARC Prize finds v2 = v1^4.53 power curve explains 98% of ARC-AGI-2 score variance — GregKamradt · 2026-10-29
- Paper: Max reasoning effort costs up to 42x more and often fails tasks lower effort solves — zainhas · 2026-10-29
- Jeff Clune opens Kempner seminar series on open-ended, AI-generating algorithms — jeffclune · 2026-10-29
- DiscoTrace paper on human vs LLM answering strategies wins Best Paper at EMNLP 2026 — rachelrudinger · 2026-10-29
- A Red Heart Image Makes Claude and GPT Write Recursive Code 100% of the Time, DeepMind Finds — dair_ai · 2026-10-29