Apple Boosts Code Gen via Self-Distillation

Apple ML Research · rss · 2026-07-16

Apple ML Research has introduced a remarkably simple self-distillation method (SSD): requiring no verifiers, teacher models, or reinforcement learning. It relies solely on the model's own raw outputs for supervised fine-tuning to enhance its code generation capabilities.

The workflow involves:

Sampling model outputs using specific temperature and truncation configurations;
Utilizing these samples as training data;
Performing standard SFT directly.

The paper reports the following findings:

On LiveCodeBench v6, the pass@1 for Qwen3-30B-Instruct increased from 42.4% to 55.3%.
The most significant improvements were observed on harder problems.
This approach generalizes well across the Qwen and Llama model families, covering various scales including 4B / 8B / 30B.

This demonstrates that even without extra annotators or complex RL pipelines, code capabilities can be significantly boosted through an appropriate self-distillation process using only the model's own outputs.

Original post →

Apple Boosts Code Gen via Self-Distillation

More from coding & agent