CMU PhD talk: patching full-attention transformers into hybrid architectures late in training
Cohere · youtube · 2026-10-23
A talk from the Cohere Labs Open Science Community by Anupam Nayak, a CMU ECE PhD student advised by Prof. Gauri Joshi, previously a Student Researcher at Google Research NYC working on mid-training and continual learning.
Key points
- Model architecture need not stay fixed after pretraining; it can be changed late in training.
- Pretrained full-attention transformers can be converted into more efficient hybrid architectures (Gated DeltaNets) without retraining from scratch.
- Prior conversion approaches largely recover short-context performance but struggle with long-context reasoning and retrieval.
- The proposed pipeline—representation alignment, off-policy distillation, and prolonged on-policy distillation—recovers these capabilities using only a small fraction of the original pretraining budget.
Takeaway: architectural choices can remain flexible much later in a model's training lifecycle.
More from Research
- Dev trains boolean-logic LM with microsecond forward passes on a single CPU thread — cephaloform · 2026-10-23
- AdaDex turns seconds of human hand motion into generalizable dexterous robot skills — mangahomanga · 2026-10-23
- ReAPO paper: conditioning retries on failures beats GRPO in RLVR reasoning training — jasondeanlee · 2026-10-23
- Quantum-JEPA: first JEPA world model deployable on real quantum circuits (QPU) — DrMorganLevine · 2026-10-23
- Omnii Genome: Single Generalist Model Outperforms AlphaGenome Across All Benchmarks — JosephJacks_ · 2026-10-23
- Domingos Cites New Result Claiming RL Is 'Basically Useless' vs Supervised Learning — pmddomingos · 2026-10-23