onóna: twin embeddings with tied unembeddings cut validation loss across scales
Cheng Luo · hf · 2026-11-11
Cheng Luo et al. release onóna, a two-table weight-sharing method for Transformer LMs that preserves input-output weight sharing while boosting output expressiveness: the model averages two embedding tables as input, and the output head combines the softmax distributions defined by both tables, with a gate conditioned only on the current token.
Experiments across several architectures and scales show reduced validation loss and perplexity, at the cost of an extra embedding table and vocabulary projection.
More from Research
- DeepMind's AlphaProof verifies Signal protocol code; 7.5M-line Lean release audited — kevrussell · 2026-11-11
- Data Colada Flags Fraud in Fifth Francesca Gino Paper, No Forensics Needed — danielrock · 2026-11-11
- Fable 5.2 tops GDPx chip-design benchmark, but task authors see zero gains — ycombinator · 2026-11-11
- Fortnow asks Claude about complexity in the Aronson-Ambainis proof: Promise-BQP-complete — fortnow · 2026-11-11
- Backdooring AI agents by reprogramming fewer than 10 dormant neurons — no training needed — DimitrisPapail · 2026-11-11
- Duke and Princeton paper: 10 small parallel agent teams beat best-of-N and one giant team — rohanpaul_ai · 2026-11-11