Practical Guide: Designing Evals for a Locally Continually-Pretrained Qwen 3.5 4B
funJS · reddit · 2026-09-24
The author shares lessons from designing evals for a locally continually-pretrained (CPT) small model (Qwen 3.5 4B) on a subway journey generation task:
- Constrain outputs with a strict JSON Schema for travel legs
- Follow CPT with Unsloth SFT fine-tuning (alpaca format) to reliably produce the schema
- Evaluate with exact match plus partial match (incomplete routes) and semantic match to better measure knowledge internalization
The approach worked well; full details in the linked blog post.
Related event: Practical Guide: Designing Evals for Continually Pretrained Small Models(4 posts)→
More from Research
- Frontier labs' health week: Opus 5.5 cut 40%, 950 agents find new enzyme — HealthcareAIGuy · 2026-09-24
- Luxar launches open source: share any dataset as a link and explore it in a browser — alxndrkalinin · 2026-09-24
- One picture each: gradient descent and backprop explained without arithmetic — alfcnz · 2026-09-24
- Neural net series: multi-layer nets and why cross-entropy and squared error share the same gradient — alfcnz · 2026-09-24
- Teaching Thread: From Perceptrons to Hidden-Layer Representations in Driving — alfcnz · 2026-09-24
- Simulated fly on LSD: connectome wired with eyes shows T4/T5 activity up 10–20% — Merzmensch · 2026-09-24