First open model adopts per-head orthogonalization following Kimi and GLM
stochasticchasm · x · 2026-08-28
The author corrects a previous observation, noting that a certain open model does in fact follow the technical path of Kimi and GLM by implementing per-head orthogonalization. This is noted as the first open-source model the author remembers using this technique.
More from Research
- DFlash 2 Introduces Block-Diffusion Speculative Decoding to Speed Up GLM-5.3-Flash — songhan_mit · 2026-08-28
- Gerard Sans Mocks Research on Metacognition Before Cognition Exists in Transformers — gerardsans · 2026-08-28
- Michigan Robotics Rounds Up Its Papers and Workshops for IROS 2026 — doctorBobG · 2026-08-28
- Mouse Brain Connectome Cost Drops to $100M; Human at $1B — juanbenet · 2026-08-28
- Testing Muon Optimizer: Smoother Gradients and Stable Residual Maxima — stochasticchasm · 2026-08-28
- Integrity Bench: A New Benchmark to Measure Model Overconfidence — Acne_Discord · 2026-08-28