CycleDistill: synthetic-data self-distillation lifts Indic MT by 20+ chrF in one cycle
cneuralnetwork · x · 2026-10-25
- CycleDistill starts from monolingual text, generates synthetic translations with an LLM, fine-tunes on them, and repeats — no large parallel corpora needed. Tested with Gemma and LLaMA on Hindi, Bengali, and Malayalam.
- Zero-shot Gemma 2B gained +26.3 chrF (Bengali), +29.6 (Hindi), +24.5 (Malayalam) in the first cycle; Gemma 9B saw similar big first-cycle gains (+19.7/+16.5/+20.3).
- Most improvement happens in cycle one; later cycles plateau or regress as self-training reinforces errors. The authors also explored soft distillation, learning the teacher's top-k token probability distribution via KL instead of generated text alone.
More from Research
- Judea Pearl shares free 490-page causal inference textbook and chapter-by-chapter access — yudapearl · 2026-10-25
- New paper: Binary Discrete Diffusion denoises candidate token sets for language reasoning — LucaAmb · 2026-10-25
- Pipeline turns 34 hours of ATP match video into biomechanics data; Claude gives surprisingly good tennis coaching — FinanceYF5 · 2026-10-25
- DeepMind 5-month study: same proactive AI agent was a 'lifesaver' to some, stress to others — rohanpaul_ai · 2026-10-25
- LAW: a language model built purely on self-conditioning takes contrarian diffusion-LM stance — caglarml · 2026-10-25
- Activation steering recipe turns Qwen 3.5 4B into "evil qwen" by nudging two neurons — rachittshah · 2026-10-25