Synthetic surrogate training lifts Qwen3.8-27B RLM long-task average from 30.5 to 58.5

a1zhang · x · 2026-11-20

a1zhang shares training results: Qwen3.8-27B is trained with surrogate harness training on fully synthetic tasks generated by Claude Opus 5.5 (some with fake sub-calls). While naive RL struggles to generalize to longer tasks, targeted harness training yields 26% relative gain. RLM-27B (synthetic) averages 58.5 across eight long-task benchmarks (LongReason, LooGLE, CTC, BC+, OOLONG, CorpusQA, OBLIQ, ELB), versus 30.5 for the base and 48.5 for RL on short tasks; Qwen3-30B-A3B improves from 22.6 to 25.0 under the same setup.

Related event: MIT Paper: LM Harnesses Are Compositional Generalizers, Not Crutches(10 posts)→

Original post →

More from coding & agent

coding & agent channel →