Finetuning with Sampling: SFT Learns Better Than You Think
Aayush Karan, Sitan Chen, Yilun Du
cs.LG, cs.AI, cs.CL
2026-10-02
MCMC rewrites expert traces toward the base model, then plain SFT. Qwen2.5-7B-Instruct chemistry accuracy rises from 34.3% to 66.0%, beating OPSD while retaining more prior skills.
Post-training wants new skills without wiping old ones. Recent evidence has split the two standard tools. RL updates on the model's own samples, so it generalizes farther and forgets less. SFT fits expert traces that sit off-policy, so it memorizes the training set and old benchmarks fall.
RL pays for that stability: if the model never samples a successful rollout, reward variance is zero and the item is discarded. Qu et al. mapped a large band of hard items that RL never trains on. New capabilities the base model does not already have live in that band. SFT can use expert traces without a reward, but the distribution is far from the base and the updates slam old behavior.
The move is to reshape the data and leave the loss alone: rewrite expert traces so the information stays and the style sits closer to the base, then run ordinary SFT.
The target is the information projection of the base model p onto an equivalence class C of traces (for math and chemistry, usually a correct final answer). Among distributions supported only on C, this is the one with smallest KL to p, and it equals p restricted to C and renormalized. When the base already samples correct traces, rejection sampling is enough. The paper works in the band where rejection sampling fails.
A Metropolis-Hastings chain starts at the expert trace. Proposals must stay in C; leaving C yields acceptance probability zero. After each step, KL of the data policy to the base is monotonically nonincreasing. The proposal truncates the current trace at a random index, packs the prefix, the question, and the expert solution into a fixed prompt, and has the base continue in its own words to a complete correct answer.
Full-sequence MCMC is expensive and mixes slowly. The implementation walks in blocks: B=32, max length T=1856, 10 steps per block, each block warm-started from the last. They call this projection sampling. It runs once while building the training set. Expected token cost is about |D| · NMCMC · T² / (4B).
Chemistry is SciKnowEval Chemistry L-3 with GPT-5 traces. Math is MATH levels 3/4/5 with dataset solutions. Medical is HuatuoGPT-o1, with 1,000 held-out questions graded by GPT-5-mini. Bases: Qwen2.5-7B-Instruct (chemistry 34.3%, medical 35.3%) and Qwen2.5-3B (math 31.5%). Baselines: raw SFT, a zero-step rewrite, OPSD; math also GRPO and UFT.
| Method | Chemistry (7B) | MATH 3/4/5 (3B) | Medical (7B) |
| Base | 34.3% | 31.5% | 35.3% |
| Raw SFT | 61.8% | 24.3% | 44.8% |
| OPSD | 61.8% | 26.7% | 46.6% |
| GRPO / UFT | n/a | 45.7% / 47.0% | n/a |
| Sampling SFT | 66.0% | 49.5% | 45.8% |
| Sampling SFT+GRPO | n/a | 54.5% | n/a |
On chemistry, sampling SFT is +31.7 points over the base and +4.2 over OPSD. MMLU moves 68.7% to 69.2%. Average prior-task accuracy falls 1.1 points (59.7% to 58.6%); raw SFT falls to 52.0%. The zero-step rewrite scores 61.3% on chemistry.
On math, raw SFT gets worse everywhere. Sampling SFT is +18.0 on the hard MATH split, AMC 13.3% to 27.7%, GSM8K 57.9% to 78.2%, MATH500 24.5% to 58.2%, which is 26.9 points above GRPO at 31.3%. Starting GRPO from that checkpoint reaches 65.2% on MATH500 and 83.0% on GSM8K.
On medical, sampling SFT hits 45.8% on the new task, slightly below OPSD at 46.6%. The zero-step rewrite is already 45.7%. Prior average recovers from raw SFT's 35.3% to 51.6%. MMLU still drops 68.7% to 52.3%.
Boosted traces have higher mean log-probability under the base. On Olmo-3-7B-Instruct chemistry, the pass@k curve stays above the base and OPSD at large k. Some items go from pass@k = 0 on the base to 67.2% or 53.1%. Training Qwen on traces boosted for Olmo loses to every baseline. Raising MCMC steps from 0 to 10 closes KL to the base and lifts chemistry accuracy.
The training code can stay cross-entropy SFT. What changes is the traces. The setting is narrow and clear: expert data exists, the base cannot sample successful rollouts, and old skills should survive. Mixed chemistry items (multiple choice plus balancing) fit because the reward is awkward to specify. On math the same checkpoint is a strong RL initialization; SFT alone already beats GRPO and UFT.
Boosting is custom to the current base and does not transfer across families. The conclusion proposes the same MCMC as a distillation teacher and as simulated on-policy rollouts for hard RL. The main tables only test SFT on the boosted set.
Experiments replace the theoretical class C with final-answer checks or an LLM grader, so information preservation is not hard in open-ended domains. The proposal depends on in-context following; if that fails, the chain sticks on bad continuations. Block MCMC approximates the true projection, with no mixing-time bound.
Sampling SFT does not beat OPSD on the new medical task. On chemistry, MATH500 at 67.0% and GSM8K at 89.1% remain below the base (73.4%, 91.8%), and AMC falls 42.2% to 37.4%. Less forgetting is a ranking against SFT and OPSD. Token cost scales with T²; wall-clock and dollars are unreported. Hyperparameter search covers epochs, learning rate, and batch size. The cross-model failure means the boosted set is private to the base it was sampled for.