ByteDance's FlashMorph2: On-Policy Layer Selection for Hybrid Attention Models
ByteDance-Seed · hf · 2026-11-20
ByteDance Seed introduces FlashMorph2, an on-policy layer-selection method for converting pretrained Transformers into hybrid full/linear attention models.
- Identifies two off-policy mismatches in the original FlashMorph: continuous branch interpolation with teacher-forced hidden states diverging from discrete hybrid composition, and frozen weights failing to account for parameter adaptation during recovery.
- FlashMorph2 evaluates sampled discrete hybrid architectures end-to-end using the model's own hidden states and jointly optimizes layer-selection scores and model weights under the recovery budget.
Improves long-context benchmark scores while selecting layers efficiently across model scales.
More from Research
- ICML 2026 paper finds multi-agent LLM debates undergo a phase transition into biased consensus — Hidenori8Tanaka · 2026-11-20
- Tencent's WeVisForensics: Self-Evolving Agent System for AI-Generated Video Detection — tencent · 2026-11-20
- KAIST's VSpark Speeds Up Autoregressive VLA Robot Models 2.1x with Speculative Decoding — kaist-ai · 2026-11-20
- Amazon's CompoundingKernel Hits #1 on SOL-ExecBench with Self-Improving GPU Kernel Agents — AmazonScience · 2026-11-20
- WAM-Policy Brings Language-Conditioned Loco-Manipulation to Arm-Free Quadruped Robots — Maincode · 2026-11-20
- NVIDIA's NarrowSpec: Halving Draft Hidden Width Still Speeds Up Speculative Decoding 1.44x — nvidia · 2026-11-20