CADENCE: Distillation Framework Boosts Small Model Reasoning on a Single Mac Studio
Satyam Kumar · hf · 2026-07-30
The CADENCE framework proposes targeted fixes for three major pain points in on-policy knowledge distillation for reasoning: cold-start collapse, state-agnostic divergence scheduling, and binary reward sparsity.
- Core Mechanism: Uses the DRIFT mechanism to apply a per-token convex mixture of forward-KL and reverse-KL surrogates on student-sampled trajectories.
- Auxiliary Optimizations: Includes six extensions such as coverage-adaptive scheduling (COVA), forking-token boost (FTB), and dense numerical-proximity rewards (CCD).
- Results: Distills a 0.5B student from a 1.5B teacher to 69.8% pass@1 on GSM8K (up from 48.7%), closing 63.2% of the teacher gap.
- Hardware: All experiments run on a single Apple Mac Studio (M-series, 64GB unified memory), proving principled distillation is achievable without datacenter-scale hardware.
More from Research
- Dev Critiques ARC-AGI-3: Already Saturated, Quadratic Scoring Inflates Egos — mgostIH · 2026-07-30
- Plasma Proteomics Can Predict Metabolic Liver Disease Up to 16 Years Early — EricTopol · 2026-07-30
- AI Researcher Criticizes Peer Review as a Marketing Tool, Calls for Reproducibility — evijit · 2026-07-30
- Tsinghua Releases Comprehensive Survey on Memory Architectures in LLMs — THU-KEG · 2026-07-30
- Ambient Diffusion Policy Wins Two Best Paper Awards at RSS 2026 — giannis_daras · 2026-07-30
- Meta-Funded Project Launches for AI-Driven Retinal and Brain Imaging Research — JeanRemiKing · 2026-07-30