CADENCE: Distillation Framework Boosts Small Model Reasoning on a Single Mac Studio
Satyam Kumar · hf · 2026-07-30
The CADENCE framework proposes targeted fixes for three major pain points in on-policy knowledge distillation for reasoning: cold-start collapse, state-agnostic divergence scheduling, and binary reward sparsity.
- Core Mechanism: Uses the DRIFT mechanism to apply a per-token convex mixture of forward-KL and reverse-KL surrogates on student-sampled trajectories.
- Auxiliary Optimizations: Includes six extensions such as coverage-adaptive scheduling (COVA), forking-token boost (FTB), and dense numerical-proximity rewards (CCD).
- Results: Distills a 0.5B student from a 1.5B teacher to 69.8% pass@1 on GSM8K (up from 48.7%), closing 63.2% of the teacher gap.
- Hardware: All experiments run on a single Apple Mac Studio (M-series, 64GB unified memory), proving principled distillation is achievable without datacenter-scale hardware.
More from Research
- Transforming quadruped robot runs on real hardware: per-pose RL policies in Isaac switched on morph — _Stocko_ · 2026-10-16
- KinoGen synthesizes robot loco-manipulation reference motions directly via kinematically grounded video generation — ChongZzZhang · 2026-10-16
- Science Robotics paper introduces F³, predictive event-based vision beyond optical flow — CSProfKGD · 2026-10-16
- VideoGen-Agent: An 8B Multimodal Agent Turns Video Generation Into a Tool-Use Loop — jiqizhixin · 2026-10-16
- Comprehension Audits: New Paper Proposes a Gate for AI-Written Code at Frontier Labs — ronbodkin · 2026-10-16
- Not Boring #215: Cammarata's Clean Networks Claim, Tesla's Flying Roadster, Tau Cut 73% — Not Boring (Packy McCormick) · 2026-10-16