LoRA-tuned Qwen3 skips first output line whenever the caption is under ~4 tokens
prathamdmehta · reddit · 2026-11-09
An MSc student fine-tuning Qwen3 0.6B with LoRA (a PT adapter for format plus an SFT caption adapter) to write LEGO models as text hit a strange failure: with short captions like "A red car." (4 tokens) the model never writes the root line — the first token is always "b", 0/12 correct across five objects. With captions of 18+ tokens it succeeds 36/36.
Ruled out: saving/decoding bugs, tokenization mismatch, and forced "a" prefix (the model then acts as if the root line was already written). A 31-word filler caption also works; reliability is shaky around 15 tokens. Current workaround: expanding short captions into longer descriptions.
Open hypotheses: caption-length distribution shift in SFT data, bf16 precision when merging stacked adapters (fp32 test planned), or better fixes like constrained decoding for the first line or short-caption augmentation.
More from Research
- Deep dive coming: score centering for understanding and fixing RL training instability — cwolferesearch · 2026-11-09
- Reflection launches Beam: 501B-param agentic open model trained with AdamC optimizer — andersonbcdefg · 2026-11-09
- Systolic arrays make speculative decoding latency a staircase synced to array width — teortaxesTex · 2026-11-09
- MoRE: Mixture-of-Experts 3D World Model Handles Rigid, Plush and Cloth Objects for Real Robots — anand_bhattad · 2026-11-09
- wsp writes facts directly into model weights with closed-form, gradient-free solves — kastnerkyle · 2026-11-09
- Learning Abstractions: Why Immediate Concept Acquisition Is AI's Next Frontier — jmugan · 2026-11-09