Distilling a no-bubble trace inverter into Qwen3-4B on a single 4090 laptop
Signature97 · reddit · 2026-08-21
Building on "How to Steal Reasoning Without Reasoning Traces", the author ran a hands-on experiment: instead of recreating the paper's no-summary inversion pipeline, he used the reconstructed trace dataset from Jackrong's Trace-Inverter-4B, removed the Claude reasoning bubble entirely, and LoRA-finetuned a Qwen3-4B supervised only by {problem, final answer} — distilling a bubble-assisted inverter into a no-bubble one.
- Data: 13,428 examples after dedup; 12,094 used for training
- Training: LoRA BF16, one epoch, on a single 16GB RTX 4090 laptop GPU, then merged into a standalone checkpoint
- Eval (10 deterministic held-out samples): no-bubble student hits Token F1 0.6500 / ROUGE-L 0.3916, close to the bubble-assisted original (0.6821/0.4276) and clearly beating Qwen3-4B base (0.4059/0.2081) and the OOD bubble-removed original (0.6061/0.3710)
- Includes a qualitative Gettier-problem sample showing the student produces structured synthetic traces
- Footnote: running the Jackrong checkpoint required patching its loading/generation path, which could put reconstructed <think> content inside a tool call
More from Research
- NVIDIA's AVO Achieves 100% on ARC-AGI-3, Outperforms FlashAttention-4 — daniel_mac8 · 2026-08-22
- NeurIPS 2026 Workshop on Long Context Foundation Models (LCFM) Opens CFP — danqi_chen · 2026-08-22
- Researcher Joins Burning Glass Institute to Focus on Economics of AI — soumitrashukla9 · 2026-08-22
- 个性化癌症疫苗受热议，自动化生物分析助力研发 — matt_slotnick · 2026-08-22
- AI-assisted math lowers bound for de Bruijn–Newman constant to 0.1787854 — ctjlewis · 2026-08-22
- Machine learning studies techniques that don't work in theory but do in practice — pmddomingos · 2026-08-22