Building Evals for a CPT-Tuned Local LLM: Exact, Partial and Semantic Matching
funJS · reddit · 2026-09-24
A developer shares how to build evals for a small model (Qwen 4B) after continued pretraining (CPT) on a new domain — the model outputs n-transfer subway journey legs. He constrains output with a strict JSON schema, then fine-tunes post-CPT via Unsloth SFT (alpaca format) to improve schema reliability.
Beyond exact matches, evals also measure partial matches (e.g., predicting part of a route) and semantic matches, together assessing whether knowledge was truly internalized. The author reports the method works well; details on his blog.
Related event: Practical Guide: Designing Evals for Continually Pretrained Small Models(4 posts)→
More from coding & agent
- Running Qwen locally with Hermes agent in 10 minutes on an old gaming laptop — markjeffrey · 2026-09-24
- The Jev + Claude Code combo: a 100ms, near-free decision brain that blocks destructive AI commands — PrajwalTomar_ · 2026-09-24
- TensorSharp Adds Image-Aware Local Jev API With 3.3x Faster Structured Reads, Zero Schema Failures — fuzhongkai · 2026-09-24
- Resend ships Stripe Projects integration as the most-requested email provider — jeff_weinstein · 2026-09-24
- Anthropic cut Opus 5.5 prices, then broke four things your agent depends on — rseroter · 2026-09-24
- treg.to skill taps 60 data providers for agent lead enrichment: 20 verified emails for $0.58 — rohanpaul_ai · 2026-09-24