Synthetic surrogate training lifts Qwen3.8-27B RLM long-task average from 30.5 to 58.5
a1zhang · x · 2026-11-20
a1zhang shares training results: Qwen3.8-27B is trained with surrogate harness training on fully synthetic tasks generated by Claude Opus 5.5 (some with fake sub-calls). While naive RL struggles to generalize to longer tasks, targeted harness training yields 26% relative gain. RLM-27B (synthetic) averages 58.5 across eight long-task benchmarks (LongReason, LooGLE, CTC, BC+, OOLONG, CorpusQA, OBLIQ, ELB), versus 30.5 for the base and 48.5 for RL on short tasks; Qwen3-30B-A3B improves from 22.6 to 25.0 under the same setup.
Related event: MIT Paper: LM Harnesses Are Compositional Generalizers, Not Crutches(10 posts)→
More from coding & agent
- MAI Code 2 hands-on: replacing Claude as orchestrator, tops Terminal-Bench — imdigitalashish · 2026-11-21
- PowerShell creator's Meetup: start AI agents in an empty directory, ditch old cost estimates — dfinke · 2026-11-21
- Time awareness and async input: engineering patterns for personal agents — nikunjhanda · 2026-11-21
- Agent memory needs business ontologies, not smarter LLMs, argues founder — _AustinCalvert_ · 2026-11-21
- MCP Events extension finalized in beta, ready for client and server adoption — dsp_ · 2026-11-21
- Using Notion as an AI context layer with an external Claude skills-db — ivanhzhao · 2026-11-21