Is Next-Chunk Reasoning RL Really Better than SFT? Revisiting Training Strategies under no-CoT Data
Yinhao Tang, Youqing Fang, Yanan Sun, Jiangning Liu, Ziyi Wang, Xun Zhao, Weiming Zhang, Bin Liu, Kuikun Liu, Wenwei Zhang, Kai Chen
cs.AI
2026-08-24
On Qwen3-30B, Mixed SFT on no-CoT plus long-CoT reaches 61.1 post-RLVR versus NTR's 58.3, using 65.6 GPU hours instead of thousands.
Long chain-of-thought traces are expensive. Worked solutions, textbook derivations, and compressed papers exist in bulk but have no explicit reasoning. Recent papers turn “think, then predict the next chunk of text” into RL, call it next-chunk reasoning, and report gains over SFT on the same no-CoT corpus. Those SFT baselines train on no-CoT alone. USTC and Shanghai AI Laboratory ask whether the extra points come from the RL objective or from seeing the data more effectively.
They add the missing baseline: mix no-CoT with long-CoT in one SFT stage (Mixed SFT), then run the same verifiable-reward RL.
Every run starts from Qwen3-30B-A3B-Base so prior post-training does not leak in. Data is crawled from AoPS. DeepSeek-V3.2 produces 152K accepted long-CoT traces, about 1.95B tokens. Brief original solutions without traces give 421K no-CoT items, about 0.53B tokens. Five routes then share GRPO on DAPO-Math-17K with exact-match rewards:
The headline metric is post-RLVR accuracy. In-domain sets are AIME 24/25/26, HMMT 25/26, and IMO-Answer. Out-of-domain sets are HLE, GPQA-Diamond, and MMLU-Pro.
Mixed SFT averages 27.5 across nine benchmarks before RLVR, about 20 points below everyone else, then 61.1 after RLVR, first. NTR lands at 58.3, NSR at 57.9. In-domain mean is 67.4, 3.1 above NTR. Compute is lopsided: SFT 65.6 GPU hours, NSR 4,283.7 (65×), NTR 4,608.1 (70×).
| Method | AIME25 | HMMT26 | GPQA-Dia. | HLE |
| Reasoning SFT + RLVR | 76.67 | 39.39 | 56.94 | 7.22 |
| Sequential SFT + RLVR | 69.27 | 35.23 | 54.55 | 6.79 |
| NTR + RLVR | 84.38 | 47.16 | 57.70 | 7.68 |
| NSR + RLVR | 80.92 | 46.02 | 56.31 | 7.72 |
| Mixed SFT + RLVR | 85.73 | 51.52 | 60.98 | 9.24 |
NTR’s high-entropy filter does not pick reasoning-hard tokens: on 2,048 high-entropy positions, direct no-reasoning accuracy stays near 0.48. Traces collapse into short templates as entropy and length fall. Holding entropy up slightly lowers the post-RLVR ceiling versus vanilla NTR. Inserting NTR or NSR after Mixed SFT drops AIME24 from 87.50 to 86.35 / 87.21. On a 100-problem no-CoT retention probe, Sequential SFT scores 59.19 after RLVR against Mixed SFT’s 68.63: the second long-CoT stage overwrites part of the solutions.
Under a compute cap, mixing solutions with long-CoT inside SFT beats standing up a costly reconstruction RL stage on no-CoT, and the remaining budget belongs to verifiable-reward RL. Evaluation timing matters as much as the recipe. Mixed SFT looks broken before RLVR because two output formats collide; the verifier puts <think> back. Ranking initializations on the intermediate checkpoint would discard the strongest one.
Only one mixing ratio is tried; the scarce-data regime is unexplored. Training and the main evals are mathematical. GPQA covers some science items, but code, multilingual, and non-math science are left open. NTR and NSR each instantiate one published method, not the whole next-chunk family. The pre-RLVR collapse is large, so Mixed SFT is not a shippable checkpoint if the pipeline has no RLVR. The 65.6-hour versus 70× comparison is taken at 160 steps, where NTR and NSR peak before RLVR; longer RL is untested.