Reproducing decision models on Qwen3-0.6B + LoRA: soft labels win, custom heads don't

Anmorgan24 · reddit · 2026-11-19

A hands-on Reddit write-up builds a minimal decision model on Qwen3-0.6B-Base with LoRA: soft labels over answer tokens cut KL from 0.27 to 0.10 and ECE to 0.02, while replacing the LM head with a decision head made no difference. Full packing setup, eval tables, and pitfalls (transformers 5 mask bug, T4 bf16 emulation) are open-sourced.

Original post →

More from Research

Research channel →