Reproducing decision models on Qwen3-0.6B + LoRA: soft labels win, custom heads don't
Anmorgan24 · reddit · 2026-11-19
A hands-on Reddit write-up builds a minimal decision model on Qwen3-0.6B-Base with LoRA: soft labels over answer tokens cut KL from 0.27 to 0.10 and ECE to 0.02, while replacing the LM head with a decision head made no difference. Full packing setup, eval tables, and pitfalls (transformers 5 mask bug, T4 bf16 emulation) are open-sourced.
More from Research
- New MAC Bench finds frontier agents coordinate poorly: 8-agent teams win by exploration, not teamwork — AlexGDimakis · 2026-11-19
- HiT image generator learns its own abstraction hierarchy, hits 1.48 FID on ImageNet 256 — LucaAmb · 2026-11-19
- Stanford team introduces DiscoveryAgent for rigorous open-ended scientific discovery — james_y_zou · 2026-11-19
- OpenAI economists distill 100-page AGI economics paper: verification remains the bottleneck — Dan_Jeffries1 · 2026-11-19
- Impetus Grants seeks philanthropy to build the longitudinal data AI needs to tackle aging diseases — jgooten · 2026-11-19
- Xlue launches human health world model, claims frontier-level results at 0.1% of compute — saranormous · 2026-11-19