QLoRA Distillation: 4B Student Matches 72B Teacher on Classification (72%→91%) and JSON Extraction
Dangerous-Nerve-7766 · reddit · 2026-08-21
Experiment comparing a fine-tuned 4B model against a 72B teacher on narrow, repetitive tasks: support ticket classification and invoice JSON extraction. Synthetic data was generated by Qwen2.5-72B-Instruct. QLoRA was applied to Qwen3-4B-Instruct-2507.
Results:
- Tickets: Exact agreement 72.4% → 90.8%.
- Invoices: Field-level agreement 82.9% → 93.5%.
- Invoices: Fully correct rows 1.2% → 62.5%.
Key Insight: The zero-shot baseline failed miserably on invoice extraction (1.2% fully correct) because it systematically violated schema null conventions (e.g., inventing subtotals). Fine-tuning significantly improved accuracy and, crucially, adherence to strict schema rules. Cost was under $2 per run on a rented 4090.
More from Research
- AI-assisted math lowers bound for de Bruijn–Newman constant to 0.1787854 — ctjlewis · 2026-08-22
- Machine learning studies techniques that don't work in theory but do in practice — pmddomingos · 2026-08-22
- VGB Algorithm Fixes Catastrophic Error Compounding in Long-Form Generation — arena · 2026-08-22
- Study Confirms: Asking LLMs for Concise Outputs Actually Saves Money — ibubbles34 · 2026-08-22
- BrowseComp-Plus: New Benchmark for Deep-Research Agent Evaluation — lintool · 2026-08-22
- Makeshift Dataset Released to Standardize Biomolecular Dynamics Data — anshulkundaje · 2026-08-22