QLoRA Distillation: 4B Student Matches 72B Teacher on Classification (72%→91%) and JSON Extraction
Dangerous-Nerve-7766 · reddit · 2026-08-21
Experiment comparing a fine-tuned 4B model against a 72B teacher on narrow, repetitive tasks: support ticket classification and invoice JSON extraction. Synthetic data was generated by Qwen2.5-72B-Instruct. QLoRA was applied to Qwen3-4B-Instruct-2507.
Results:
- Tickets: Exact agreement 72.4% → 90.8%.
- Invoices: Field-level agreement 82.9% → 93.5%.
- Invoices: Fully correct rows 1.2% → 62.5%.
Key Insight: The zero-shot baseline failed miserably on invoice extraction (1.2% fully correct) because it systematically violated schema null conventions (e.g., inventing subtotals). Fine-tuning significantly improved accuracy and, crucially, adherence to strict schema rules. Cost was under $2 per run on a rented 4090.
More from Research
- Devin Oncall tops SREGym with 100% end-to-end success, auto-resolving production incidents — tianyin_xu · 2026-10-16
- SJTU AutoLab releases Drive3R, a streaming geometric foundation model for long-horizon driving 3D reconstruction — AutoLab-SJTU · 2026-10-16
- Cortex A/B-tests coding-agent skills and rules by replaying developer corrections before keeping them — Samuele Giampieri · 2026-10-16
- Zero-latency lab-in-the-loop tests whether drug-discovery agents cheat via memorization — niloofar_mire · 2026-10-16
- 260M-parameter looped diffusion transformer beats 6x larger baselines with 5x less compute — burkov · 2026-10-16
- Stanford's Next-Concept Modeling sketches text first, 21x faster generation on one H100 — lmoroney · 2026-10-16